Commit Graph

70 Commits

Author SHA1 Message Date
Romil Bhardwaj 0f42f87ebc Updating zero capacity resource semantics (#4555) 2019-04-12 16:53:57 -07:00
Wang Qing 5cfbfe5df6 [Java] Implement GcsClient (#4601) 2019-04-12 22:44:47 +08:00
Wang Qing fe07a5b4b1 Add delete_creating_tasks option for internal.free() (#4588)
* add delete creating task objects.

* format code style

* Fix lint

* add tests add address comments.

* Refine test

* Refine java test

* Fix CI

* Refine

* Fix lint

* Fix CI
2019-04-12 13:38:31 +08:00
Yuhong Guo c2349cf12d Remove local/global_scheduler from code and doc. (#4549) 2019-04-03 17:05:09 -07:00
bibabolynn ab55a1f93a [Java] Clean up outdated dependencies (#4489) 2019-03-28 14:33:45 +08:00
bibabolynn 7a9d1546d4 [java] Fix getWorker and add create multi actors test (#4472) 2019-03-26 20:26:13 +08:00
Wang Qing 7d70cfba6e [Java] Fix loading custom classes from jars (#4475) 2019-03-26 20:15:08 +08:00
Ruifang Chen 59d74d5e92 [Java] Build Java code with Bazel (#4284) 2019-03-22 14:30:05 +08:00
Hao Chen d03999d01e Cross-language invocation Part 1: Java calling Python functions and actors (#4166) 2019-03-21 13:34:21 +08:00
Hao Chen a6a5b344b9 [Java] Upgrade checkstyle plugin (#4375) 2019-03-15 11:36:09 -07:00
Hao Chen f8d12b0418 [Java] Package native dependencies into jar (#4367) 2019-03-15 12:38:40 +08:00
Yuhong Guo d5fb7b70a9 Update arrow version to fix plasma bugs (#4127)
* Update arrow

* Change to 2c511979b13b230e73a179dab1d55b03cd81ec02 which is rebased on Arrow 46f75d7

* Update to fix comment

* disable tests which use python/ray/rllib/tests/data/cartpole_small

* Fix get order of meta and data in MockObjectStore.java
2019-03-08 18:03:58 +08:00
Hao Chen f0465bc68c [Java] Refine tests and fix single-process mode (#4265) 2019-03-07 09:59:13 +08:00
Wang Qing a116b7f646 [Java] Add runtime context (#4194) 2019-03-05 20:25:29 +08:00
bibabolynn c73d5086f3 [Java] Single-process mode (#4245) 2019-03-05 13:50:20 +08:00
Philipp Moritz 9b3ce3e64b Revert inline objects PR (#4125)
* Revert "Inline objects (#3756)"

This reverts commit f987572795.

* fix rebase problems

* more rebase fixes

* add back debug statement
2019-02-22 18:21:01 -08:00
Hao Chen a99676e39b [Java] lint unused imports (#4100) 2019-02-20 12:37:04 -08:00
Hao Chen de17443dc2 Propagate backend error to worker (#4039) 2019-02-16 11:39:15 +08:00
Hao Chen f31a79f3f7 Implement actor checkpointing (#3839)
* Implement Actor checkpointing

* docs

* fix

* fix

* fix

* move restore-from-checkpoint to HandleActorStateTransition

* Revert "move restore-from-checkpoint to HandleActorStateTransition"

This reverts commit 9aa4447c1e3e321f42a1d895d72f17098b72de12.

* resubmit waiting tasks when actor frontier restored

* add doc about num_actor_checkpoints_to_keep=1

* add num_actor_checkpoints_to_keep to Cython

* add checkpoint_expired api

* check if actor class is abstract

* change checkpoint_ids to long string

* implement java

* Refactor to delay actor creation publish until checkpoint is resumed

* debug, lint

* Erase from checkpoints to restore if task fails

* fix lint

* update comments

* avoid duplicated actor notification log

* fix unintended change

* add actor_id to checkpoint_expired

* small java updates

* make checkpoint info per actor

* lint

* Remove logging

* Remove old actor checkpointing Python code, move new checkpointing code to FunctionActionManager

* Replace old actor checkpointing tests

* Fix test and lint

* address comments

* consolidate kill_actor

* Remove __ray_checkpoint__

* fix non-ascii char

* Loosen test checks

* fix java

* fix sphinx-build
2019-02-13 19:39:02 +08:00
Wang Qing c523bc04ad Enable redis password in Java worker (#3943)
* Support Java redis password

* Fix

* Refine

* Fix lint.
2019-02-12 13:11:25 +08:00
Wang Qing bc438ca73b [Java] Refine Java config item (#4014)
* Refine

* Address comment.
2019-02-11 23:55:40 +08:00
Yuhong Guo 3a66d47a3a Remove RAY_CHECK from JNI code (#3978)
* Remove RAY_CHECK in JNI

* Try to add mvn test to test the exception.

* Refine

* Address comments
2019-02-09 18:10:22 +08:00
Ion f987572795 Inline objects (#3756)
* added store_client_ to object_manager and node_manager

* half through...

* all code in, and compiling! Nothing tested though...

* something is working ;-)

* added a few more comments

* now, add only one entry to the in GCS for inlined objects

* more comments

* remove a spurious todo

* some comment updates

* add test

* added support for meta data for inline objects

* avoid some copies

* Initialize plasma client in tests

* Better comments. Enable configuring nline_object_max_size_bytes.

* Update src/ray/object_manager/object_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* Update src/ray/raylet/node_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* Update src/ray/raylet/node_manager.cc

Co-Authored-By: istoica <istoica@cs.berkeley.edu>

* fiexed comments

* fixed various typos in comments

* updated comments in object_manager.h and object_manager.cc

* addressed all comments...hopefully ;-)

* Only add eviction entries for objects that are not inlined

* fixed a bunch of comments

* Fix test

* Fix object transfer dump test

* lint

* Comments

* Fix test?

* Fix test?

* lint

* fix build

* Fix build

* lint

* Use const ref

* Fixes, don't let object manager hang

* Increase object transfer retry time for travis?

* Fix test

* Fix test?

* Add internal config to java, fix PlasmaFreeTest
2019-02-07 10:32:39 -08:00
Philipp Moritz 3bb65677dc Use one memory mapped file for plasma (#3871) 2019-02-06 23:53:05 -08:00
Wang Qing e1c68a0881 Enable including Java worker for ray start command (#3838) 2019-02-04 16:23:43 +08:00
Wang Qing dcb744518e Implement actor dummy object gc in java (#3822)
* Add dummy object gc in java

* Fix

* Address comments.

* Refine

* Address comments.
2019-01-23 11:56:25 -08:00
Wang Qing 816406ea3d [Java] Fix setCurrentTask() in multi threading (#3821) 2019-01-23 20:45:30 +08:00
Wang Qing 3cf59855af [Java] Replace junit with testNG (#3768) 2019-01-14 17:49:17 +08:00
Hao Chen 1bb20badec [Java] Fix bug when actor creation task fails (#3740)
* [Java] Fix bug when actor creation task fails

* remove imports
2019-01-14 11:09:15 +08:00
Wang Qing 8674606e26 Support to auto-generate Java files from flatbuffer (#3749)
* auto gen flatbuffers for Java

* Add auto_gen_tool.py

* Refine

* Add a comment

* address comments.

* Address comments.

* Addressed

* Refine

* Address comments

* Fix typo

* Add exception

* Address comments.

* Refine

* Fix lint

* Fix

* Fix lint and address comment.

* Fix lint error
2019-01-13 11:39:23 -08:00
Wang Qing 0a556dc0b5 Refine redis client (#3758) 2019-01-12 23:01:48 +08:00
Wang Qing a0cf8ee5a8 Refine Java worker code (#3735) 2019-01-12 22:45:33 +08:00
Hao Chen 597abb24ea Refine multi-threading support (#3672)
* [Python] refine multi-threading support

fix

* [java] refine multithreading code

fix java

* format
2019-01-10 13:58:11 -08:00
Stephanie Wang 04f31db54d Actor dummy object garbage collection (#3593)
* Convert UniqueID::nil() to a constructor

* Cleanup actor handle pickling code

* Add new actor handles to the task spec

* Pass in new actor handles

* Add new handles to the actor registration

* Regression test for actor handle forking and GC

* lint and doc

* Handle pickled actor handles in the backend and some refactoring

* Add regression test for dummy object GC and pickled actor handles

* Check for duplicate actor tasks on submission

* Regression test for forking twice, fix failed named actor leak

* Fix bug for forking twice

* lint

* Revert "Fix bug for forking twice"

This reverts commit 3da85e59d401e53606c2e37ffbebcc8653ff27ac.

* Add new actor handles when task is assigned, not finished

* Remove comment

* remove UniqueID()

* Updates

* update

* fix

* fix java

* fixes

* fix
2019-01-09 10:37:11 -08:00
Wang Qing 692fdc6bc3 [Java] Allow actor handle to be serialized without forking (#3686) 2019-01-06 00:29:08 +08:00
Wang Qing c59b506c6e [Java] Support calling Ray APIs from multiple threads (#3646) 2018-12-28 17:44:31 +08:00
Wang Qing 4cde971916 [Java] Print the log message slowly. (#3633) 2018-12-26 16:33:21 +08:00
Wang Qing a971b73bbe [Java] Fix the issue when waiting an empty list or a null pointer (#3632) 2018-12-26 11:29:29 +08:00
Wang Qing 8393df2516 Use BaseTest to instead of TestListener. (#3577) 2018-12-21 16:29:16 -08:00
bibabolynn e65b8f18f4 [java] change RayLog.core to org.slf4j.Logger (#3579) 2018-12-21 15:58:32 +08:00
Yuhong Guo fb33fa9097 Enable function_descriptor in backend to replace the function_id (#3028) 2018-12-18 18:53:59 -05:00
bibabolynn 7fd24e384b [java] Pass large args by reference (#3504) 2018-12-14 23:32:35 +08:00
Yuhong Guo a4abe6c0fe Add test to test raylet client connection when raylet crashes. (#3518) 2018-12-13 23:40:50 -08:00
Hao Chen e7b51cbd1b [xray] Implement Actor Reconstruction (#3332)
* Implement Actor Reconstruction

* fix

* fix actor handle __del__

* fix lint

* add comment

* Remove actorCreationDummyObjectId

* address comments

* fix

* address comments

* avoid copy

* change log to debug

* fix error name
2018-12-13 21:28:58 -08:00
Si-Yuan 84fae57ab5 Convert the raylet client (the code in local_scheduler_client.cc) to proper C++. (#3511)
* refactoring

* fix bugs

* create client class

* create client class for java; bug fix

* remove legacy code

* improve code by using std::string, std::unique_ptr rename private fields and removing legacy code

* rename class

* improve naming

* fix

* rename files

* fix names

* change name

* change return types

* make a mutex private field

* fix comments

* fix bugs

* lint

* bug fix

* bug fix

* move too short functions into the header file

* Loose crash conditions for some APIs.

* Apply suggestions from code review

Co-Authored-By: suquark <suquark@gmail.com>

* format

* update

* rename python APIs

* fix java

* more fixes

* change types of cpython interface

* more fixes

* improve error processing

* improve error processing for java wrapper

* lint

* fix java

* make fields const

* use pointers for [out] parameters

* fix java & error msg

* fix resource leak, etc.
2018-12-13 13:39:10 -08:00
Yuhong Guo 0136af5aac Add return value for recontruction RPC. (#3493)
* Add return value for recontruct RPC.

* Fix comment function name
2018-12-09 00:08:44 -08:00
Hao Chen abd37df41e Add stress test for Java worker (#3424) 2018-12-01 16:11:09 -08:00
Stephanie Wang 447604a9fe Use actor ID for the dummy object (#3437) 2018-11-29 22:31:04 -08:00
Stephanie Wang d950e92f63 Allow multiple threads to call ray.get and ray.wait (#3244)
* Handle multiple threads calling ray.get

* Multithreaded ray.wait

* Pass in current task ID in java backend

* Add multithreaded actor to tests, add warning messages to worker for multithreaded ray.get

* Fix test

* Some cleanups

* Improve error message

* Add assertion

* Cleanup, throw error in HandleTaskUnblocked if task not actually blocked

* lint

* Fix python worker reset

* Fix references to reconstruct_objects

* Linting

* java lint

* Fix java

* Fix iterator
2018-11-07 22:39:28 -08:00
Richard Liaw 0bab8ed95c Expose internal config parameters for starting Ray (#3246)
## What do these changes do?

This PR exposes the CL option for using a config parameter. This is important for certain tests (i.e., FT tests that removing nodes) to run quickly.

Note that this is bad practice and should be replaced with GFLAGS or some equivalent as soon as possible.

#3239 depends on this.

TODO:
 - [x] Add documentation to method arguments before merging.
 - [x] Add test to verify this works?

## Related issue number
2018-11-07 21:46:02 -08:00