Commit Graph
55 Commits
Author SHA1 Message Date
songqingandPhilipp Moritz 4dd4698564 unify build dir for Python and Java (#2171)
* unify build dir for Python and Java

* enable executables auto installed when just running 'make'

* fix plasma_store copy error

* fix cmake error about copying executables

* lint fix

* recover python/setup.py

* enable to copy optional file automatically

* a small fix of path

* lint fix

* lint fix

* lint fix

* Add comment.
2018-06-01 16:28:27 -07:00
Yujie LiuandPhilipp Moritz a8d3c057c1 [JavaWorker] Enable java worker support (#2094)
* Enable java worker support
--------------------------
This commit includes a tailored version of the Java worker implementation from Ant Financial.
The changes for build system, python module, src module and arrow are in other commits, this commit consists of the following modules:
 - java/api: Ray API definition
 - java/common: utilities
 - java/hook: binary rewrite of the Java byte-code for remote execution
 - java/runtime-common: common implementation of the runtime in worker
 - java/runtime-dev: a pure-java mock implementation of the runtime for fast development
 - java/runtime-native: a native implementation of the runtime
 - java/test: various tests

Contributors for this work:
 Guyang Song, Peng Cao, Senlin Zhu,Xiaoying Chu, Yiming Yu, Yujie Liu, Zhenyu Guo

* change the format of java help document from markdown to RST

* update the vesion of Arrow for java worker

* adapt the new version of plasma java client from arrow which use byte[] instead of custom type

* add java worker test to ci

* add the example module for better usage guide
2018-05-26 14:38:50 -07:00
Zongheng YangandRobert Nishihara fa97acbc89 Integrate credis with Ray & route task table entries into credis. (#1841) 2018-05-24 23:35:25 -07:00
Philipp MoritzandRobert Nishihara 225608ec66 Update arrow to latest master (#2100) 2018-05-24 00:26:13 -07:00
Philipp MoritzandRobert Nishihara af88fdefcf Incorporate C++ Buffer management and Seal global threadpool fix from arrow (#1950) 2018-04-25 22:53:44 -07:00
Robert NishiharaandPhilipp Moritz 24c944e499 Update arrow to efficiently serialize more types of numpy arrays. (#1889) 2018-04-13 09:41:51 -07:00
Robert NishiharaandPhilipp Moritz d0fffec2d0 Update arrow and parquet-cpp. (#1875)
* Update arrow.

* Fix bug.

* Cherry-pick commit for fixing parquet segfault.

* Update arrow and revert auto-releasing buffer commit.

* Remove parquet cherry-pick.
2018-04-12 16:17:12 -07:00
alonamidandRobert Nishihara 202f9683ea check if arrow build dir exists (#1863) 2018-04-10 14:52:51 -07:00
Robert NishiharaandPhilipp Moritz d8adfbd440 Fail build if pyarrow hasn't successfully been built. (#1721) 2018-03-16 12:02:36 -07:00
Philipp MoritzandRobert Nishihara 5ef0892236 Compile boost from source to fix macOS wheels (#1688) 2018-03-08 23:22:23 -08:00
Philipp MoritzandRobert Nishihara a9acfab3a6 Start chain replicated GCS with Ray (#1538) 2018-03-07 10:18:58 -08:00
Zhenyu GuoandRobert Nishihara f1e5789c26 restructure how to organize 3rd party libs (#1630)
* restructure how to organize 3rd party libs

* Minor whitespace changes.

* Fix compilation on Linux.

* Pass around Python executable so that the correct version of Python is used.
2018-03-01 14:29:56 -08:00
mehrdadnandRobert Nishihara 7237ec4124 Windows compatibility (#57)
* Add Python and Redis submodules, and remove old third-party modules

* Update VS projects (WARNING: references files that do not exist yet)

* Update code & add shims for APIs except AF_UNIX/{send,recv}msg()

* Minor style changes.
2016-11-22 17:04:24 -08:00
Robert NishiharaandPhilipp Moritz f83a98d71a Changes to make tests pass on Travis. (#3)
* Remove hiredis submodule.

* Squashed 'src/common/thirdparty/hiredis/' content from commit acd1966

git-subtree-dir: src/common/thirdparty/hiredis
git-subtree-split: acd1966bf7f5e1be74b426272635c672def78779

* Make Plasma tests pass.

* Make Photon tests pass.

* Compile and test with Travis.

* Deactive fetch test so that the tests pass.
2016-10-25 22:39:21 -07:00
Robert Nishihara 7c1b2f702f Remove git submodules and C++ files. 2016-10-25 12:51:30 -07:00
mehrdadnandPhilipp Moritz c6ec36865d Update Windows patches for GRPC (#444) 2016-10-06 11:50:40 -07:00
Robert NishiharaandPhilipp Moritz de6ec47f9e Add a recursion depth for serialization to prevent infinite loops. (#440) 2016-09-19 17:17:42 -07:00
Robert NishiharaandPhilipp Moritz 91f16a3df0 Migrate repositories to ray-project. (#438)
* Migrate repositories to ray-project.

* Update numbuf to the migrated version.
2016-09-17 00:52:05 -07:00
Robert NishiharaandPhilipp Moritz 1aa89a4ae6 Update numbuf to properly handle Python floats. (#435) 2016-09-15 15:44:11 -07:00
Robert NishiharaandPhilipp Moritz d264713ceb Work around Arrow bug by increasing metadata size. (#415) 2016-09-07 18:46:04 -07:00
Robert NishiharaandPhilipp Moritz 11a8914684 Allow users to serialize custom classes. (#393)
* Allow serialization of custom classes.

* Add documentation and test cases, also fix pickle case.

* Don't allow old-style classes.
2016-09-06 13:28:24 -07:00
Philipp MoritzandRobert Nishihara 68cec55a98 Refcount without modifying objects (#407)
* refcount without modifying objects

* add documentation

* Update tests and documentation.

* Remove extraneous code.

* Update numbuf version.
2016-09-04 12:07:52 -07:00
Stephanie WangandPhilipp Moritz b6872182bf Add hiredis to build (#395)
* Add hiredis to build

* make hiredis build if ray has been cloned before this commit
2016-09-01 20:15:00 -07:00
Philipp MoritzandRobert Nishihara 93e6c9947b update numbuf (#392)
* update numbuf

* Augment serialization tests.
2016-08-25 20:05:48 -07:00
Robert NishiharaandPhilipp Moritz 43452b9ab3 Try grpc version 1.0 (#386) 2016-08-23 19:29:18 -07:00
Robert NishiharaandPhilipp Moritz 87bb7a8f67 [WIP] Large changes to make the tests pass. (#376)
* Revert "Make tests more informative (#372)"

This reverts commit fd353250c8.

* fix bugs, in particular deactivate worker service on driver and remove condition variables

* changes to minimize the changes in this PR

* switch from faulty mutex synchronization to using atomics

* Increase the default size of the message queues, to accommodate exporting large numbers of remote functions. This is a temporary fix, but not a long term solution.

* Reorganize the scheduler export code to queue up exports. This does not solve the underlying problem yet, but sets up a solution.

* Start a separate thread on driver to print error messages by constantly querying the scheduler. This is a temporary solution because the solution based on starting a worker service for the driver which the scheduler can push error messages to is buggy.

* Fix segfault in taskcapsule destructor.

* Move tests for catching errors into a separate test file.

* Revert "roll back grpc (#368)"

This reverts commit c01ef95d04.
2016-08-15 11:02:54 -07:00
Philipp MoritzandRobert Nishihara c01ef95d04 roll back grpc (#368) 2016-08-10 16:45:40 -07:00
Philipp MoritzandJohann Schleier-Smith 9903fc0335 Improve error message for serialization of custom objects (#359)
* Improve error message for serialization of custom objects

* gitignore numbuf build directory
2016-08-09 13:38:21 -07:00
Philipp MoritzandRobert Nishihara 8bf877ac1e Serialize and Deserialize unicode (#349) 2016-08-04 21:06:31 -07:00
mehrdadnandPhilipp Moritz 1961deeffb Update Windows support (#317) 2016-07-28 13:11:13 -07:00
mehrdadnandPhilipp Moritz 82480135a6 numbuf (#316) 2016-07-28 11:42:39 -07:00
mehrdadnandPhilipp Moritz c3695099fc numbuf update (#313) 2016-07-27 23:21:33 -07:00
mehrdadnandPhilipp Moritz 65db47b556 Update submodules (#309) 2016-07-27 19:47:14 -07:00
Philipp MoritzandRobert Nishihara 2d929ad938 Only check out submodules that are actually used (#310) 2016-07-27 19:14:40 -07:00
Philipp MoritzandRobert Nishihara 5591aa4665 cleanup serialization code (#291) 2016-07-25 15:47:10 -07:00
Philipp MoritzandRobert Nishihara 4a0f35b042 new arrow serialization code (serialize python objects recursively) (#284) 2016-07-25 13:41:47 -07:00
mehrdadnandRobert Nishihara c5be9b96de Visual Studio - Project cleanup, warning suppression, and removal of zlib dependency (#272)
* windows_intermodule_singleton warning

* Fix some warnings from Visual Studio

* More Visual Studio project changes

* Replace zlib Visual Studio project with stub; it's not needed

* Un-enable link-time code generation
2016-07-16 13:02:51 -07:00
mehrdadnandPhilipp Moritz 69ee25eb51 Fix GRPC project patch, again (#255) 2016-07-10 22:31:12 -07:00
mehrdadnandPhilipp Moritz 30023b456a Fix GRPC project patch (#254) 2016-07-10 22:05:50 -07:00
mehrdadnandPhilipp Moritz 5db31cd8a2 Update numbuf (#246) 2016-07-09 21:47:53 -07:00
mehrdadnandPhilipp Moritz 302591cc0b Edit batch file again (#241) 2016-07-09 21:23:02 -07:00
mehrdadnandRobert Nishihara fc711abda0 Remove parquet for Windows -- we don't need it yet (#240) 2016-07-08 20:06:21 -07:00
mehrdadnandRobert Nishihara 7ff1580606 Apply patches while considering index as well (#231) 2016-07-08 09:24:25 -07:00
mehrdadnandRobert Nishihara 63f7548763 Fix wrong branch to checkout (and do recursively if possible) (#226) 2016-07-07 00:22:36 -07:00
mehrdadnandRobert Nishihara eff6924b7c Update third-party patches (#203) 2016-07-05 12:17:09 -07:00
Philipp MoritzandRobert Nishihara 72e1b0fcd6 fix warnings (#207) 2016-07-04 15:40:56 -07:00
Philipp MoritzandRobert Nishihara 3eeb2d315c [WIP] update grpc (#206)
* update grpc

* use GRPC's OpenSSL library
2016-07-04 14:24:28 -07:00
Philipp Moritz b99ea3ca4e build ray in parallel 2016-06-25 15:35:52 -07:00
Philipp Moritz 8a0b8e8789 Unmap segments properly to avoid memory leaks 2016-06-24 16:56:26 -07:00
mehrdadnandPhilipp Moritz fe918a9f8b Update numbuf to master (#147) 2016-06-22 16:33:04 -07:00