b89cac976a
Basic direct actor call support in Python ( #5991 )
2019-10-28 22:09:04 -07:00
c1418b04df
Remove CoreWorkerObjectInterface ( #6023 )
2019-10-28 10:48:41 -07:00
80c01617a3
Optimize python task execution ( #6024 )
2019-10-27 00:43:34 -07:00
a5523466a2
Enable memstore by default ( #6003 )
2019-10-25 21:59:12 -07:00
d4055d70e3
Remove CoreWorkerTaskExecutionInterface ( #6009 )
2019-10-25 16:33:44 -07:00
e6141a0b8b
Remove UsePush logic from raylet ( #6015 )
2019-10-25 14:52:19 -07:00
1ce521a7f3
Remove task context from python worker ( #5987 )
...
Removes duplicated state between the python and C++ workers. Also cleans up the serialization codepaths a bit.
2019-10-25 07:38:33 -07:00
71a2f4c63d
fix comment ( #6006 )
2019-10-24 18:07:49 -07:00
c73fdb7425
Ignore errors in ObjectID.__dealloc__ ( #5997 )
2019-10-24 16:48:47 -07:00
09d05bb3fa
Reduce actor submission python overhead ( #5949 )
2019-10-23 00:11:32 -07:00
02931e08f3
[core worker] Python core worker task execution ( #5783 )
...
Executes tasks via the event loop in the C++ core worker. Also properly handles signals (including KeyboardInterrupt), so ctrl-C in a python interactive shell works now (if connecting to an existing cluster).
2019-10-22 20:15:59 -07:00
fc56872012
Send active object IDs to the raylet ( #5803 )
...
* Send active object IDs to the raylet
* comment
* comments
* dedup
* signed int in config
* comments
* Remove object ID from monitor
* Fix test
* re-add check
* fix cast
* check if core worker
* Add comment
* Reservoir sampling
* Fix lint
* Pointer return
* tmp
* Fix merge
* Initialize object ids properly
* Fix lint
2019-10-20 22:05:28 -07:00
697f765efc
Refactor CoreWorker to remove TaskInterface ( #5924 )
...
* Remove TaskInterface
* Remove Status return value
* Remove CActorHandle, some return values, TaskSubmitter
* lint
* doc
* doc
* fix build
* lint
* Return Status, guarded by annotation, fail tasks for RECONSTRUCTING actors
* fix
* move annotation
* revert
* Fix core worker test
* nits
2019-10-18 00:03:57 -04:00
3ac8592dcf
Remove actor handle IDs ( #5889 )
...
* Remove actor handle ID from main ActorHandle constructor
* Set the actor caller ID when calling submit task instead of in the actor handle
* Remove ActorHandle::Fork, remove actor handle ID from protobuf
* Make inner actor handle const, remove new_actor_handles
* Move caller ID into the common task spec, start refactoring raylet
* Some fixes for forking actor handles
* Store ActorHandle state in CoreWorker, only expose actor ID to Python
* Remove some unused fields
* lint
* doc
* fix merge
* Remove ActorHandleID from python/cpp
* doc
* Fix core worker test
* Move actor table subscription to CoreWorker, reset actor handles on actor failure
* lint
* Remove GCS client from direct actor
* fix tests
* Fix
* Fix tests for raylet codepath
* Fix local mode
* Fix multithreaded test
* Fix AsyncSubscribe issue...
* doc
* fix serve
* Revert bazel
2019-10-17 12:36:34 -04:00
42dd0fae96
Fix actor ID collision in local mode ( #5863 )
...
* Fixed local mode actor id
* Update python/ray/actor.py
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com >
* Added hyphen to match comments
* Added tests to test_local_mode
* Helloworld
* Better test naming
* lint
2019-10-08 13:07:42 -07:00
08e4e3a153
[core worker] Submit Python actor tasks through core worker ( #5750 )
...
* Submit actor tasks through core worker
* Fix java
* add comment
* Remove task builder
* Check negative
* Increase -> Increment
* pass by reference
* fix signal
* Clean up c++ actor handle
* more cleanup
* Clean up headers
* Fix unique_ptr construction
* Fix java
* Move profiling to c++
* dedup
* fix error
* comments
* fix java
* Fix tests
* wait for actor to exit
* Start after constructor
* ignore java build
* fix comment
* always init logging
* Fix logging
* fix logging issue
* shared_ptr for profiler
* DEBUG -> WARNING
* fix killed_ init
* Fix flaky checkpointing tests
* -v flag for tune tests
* Fix checkpoint test logic
* Fix exception matching
* timeout exception
* Fix test exception info
* Fix import
* fix build
* Fix test
* shared_ptr
2019-10-07 15:42:19 -07:00
3a42780cb8
Improved Pickle5 pickling ( #5841 )
...
* object copy optimization
* see if we can reuse the Arrow parallel_memcopy
* remove unused function
* restore the original code, since later experiments show that it has little impact on performance.
* lint
2019-10-03 15:14:32 -07:00
2fb7d7846f
Initial implementation of Cython pickle5 support ( #5725 )
2019-10-03 09:20:26 -07:00
963bbe8bbd
Move profiling to c++ ( #5771 )
...
* Move profiling to c++
* comments
* Fix tests
* Start after constructor
* fix comment
* always init logging
* Fix logging
* fix logging issue
* shared_ptr for profiler
* DEBUG -> WARNING
* fix killed_ init
* Fix flaky checkpointing tests
* Fix checkpoint test logic
* Fix exception matching
* timeout exception
* Fix import
* fix build
* use boost::asio
* fix double const
* Properly reset async_wait
* remove SIGINT
* Change error message
* increase timeout
* small nits
* Don't trap on SIGINT
* -v for tune
* Fix test
2019-10-01 10:06:25 -07:00
61e5d674be
Push driver task in core worker ( #5752 )
2019-09-23 10:53:55 -05:00
a5d7de6aaf
[core worker] Python core worker normal task submission ( #5566 )
2019-09-14 13:02:53 -07:00
07c4c6367a
[core worker] Python core worker object interface ( #5272 )
2019-09-12 23:07:46 -07:00
ed761900f6
[Java] Support direct actor call in Java worker ( #5504 )
2019-09-09 14:29:20 +08:00
d8f5804690
Support metadata for passing by value task arguments ( #5527 )
2019-09-08 11:07:48 +08:00
732336fc4f
[Java] Support multiple workers in Java worker process ( #5505 )
2019-09-07 22:52:05 +08:00
bea43c85b1
Ref count objects created with ray.put ( #5590 )
...
* wip
* add weakref option
* tune util
* test ref
* doc
* fix tests
2019-08-30 16:43:23 -07:00
e2e30ca507
Ray, Tune, and RLlib support for memory, object_store_memory options ( #5226 )
2019-08-21 23:01:10 -07:00
599cc2be60
Revert raylet to worker GRPC communication back to asio ( #5450 )
2019-08-17 19:11:32 -07:00
d372f24e3c
[ID Refactor] Refactor ActorID, TaskID and ObjectID ( #5286 )
...
* Refactor ActorID, TaskID on the Java side.
Left a TODO comment
WIP for ObjectID
ADD test
Fix
Add java part
Fix Java test
Fix
Refine test.
Enable test in CI
* Extra a helper function.
* Resolve TODOs
* Fix Python CI
* Fix Java lint
* Update .travis.yml
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Address some comments.
Address some comments.
Add id_specification.rst
Reanme id_specification.rst to id_specification.md
typo
Address zhijun's comments.
Fix test
Address comments.
Fix lint
Address comments
* Fix test
* Address comments.
* Fix build error
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Address comments
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/common/id.h
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update src/ray/design_docs/id_specification.md
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Address comments.
* Address comments.
* Address comments.
* Update C++ part to make sure task id is generated determantic
* WIP
* Fix core worker
* Fix Java part
* Fix comments.
* Add Python side
* Fix python
* Address comments
* Fix linting
* Fix
* Fix C++ linting
* Add JobId() method to TaskID
* Fix linting
* Update src/ray/common/id.h
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/ActorId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Address comments
* Add DriverTaskId embeding job id
* Fix tests
* Add python dor_fake_driver_id
* Address comments and fix linting
* Fix CI
2019-08-07 11:04:51 +08:00
40395acadf
[gRPC] Migrate raylet client implementation to grpc ( #5120 )
2019-07-25 14:48:56 +08:00
e5be5fd46d
Remove dependencies from TaskExecutionSpecification ( #5166 )
2019-07-15 18:15:21 -07:00
fd835d107e
Move task to common module and add checks in getter methods ( #5147 )
2019-07-11 17:07:04 +08:00
f2293243cc
[ID Refactor] Shorten the length of JobID to 4 bytes ( #5110 )
...
* WIP
* Fix
* Add jobid test
* Fix
* Add python part
* Fix
* Fix tes
* Remove TODOs
* Fix C++ tests
* Lint
* Fix
* Fix exporting functions in multiple ray.init
* Fix java test
* Fix lint
* Fix linting
* Address comments.
* FIx
* Address and fix linting
* Refine and fix
* Fix
* address
* Address comments.
* Fix linting
* Fix
* Address
* Address comments.
* Address
* Address
* Fix
* Fix
* Fix
* Fix lint
* Fix
* Fix linting
* Address comments.
* Fix linting
* Address comments.
* Fix linting
* address comments.
* Fix
2019-07-11 14:25:16 +08:00
8a30b93e42
Define common data structures with protobuf. ( #5121 )
2019-07-08 22:41:37 +08:00
62e4b591e3
[ID Refactor] Rename DriverID to JobID ( #5004 )
...
* WIP
WIP
WIP
Rename Driver -> Job
Fix complition
Fix
Rename in Java
In py
WIP
Fix
WIP
Fix
Fix test
Fix
Fix C++ linting
Fix
* Update java/runtime/src/main/java/org/ray/runtime/config/RayConfig.java
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/core_worker/core_worker.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Address comments
* Fix
* Fix CI
* Fix cpp linting
* Fix py lint
* FIx
* Address comments and fix
* Address comments
* Address
* Fix import_threading
2019-06-28 00:44:51 +08:00
d106283769
Better organize ray_common module ( #4898 )
2019-06-04 23:19:09 -07:00
1f0809e2b4
Refactor ID Serial 2: change all ID functions to CamelCase ( #4896 )
2019-05-31 11:31:18 +08:00
1a39fee9c6
Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID ( #4776 )
...
* Enable BaseId.
* Change TaskID and make python test pass
* Remove unnecessary functions and fix test failure and change TaskID to
16 bytes.
* Java code change draft
* Refine
* Lint
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/ObjectId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Address comment
* Lint
* Fix SINGLE_PROCESS
* Fix comments
* Refine code
* Refine test
* Resolve conflict
2019-05-22 14:46:30 +08:00
004440f526
Dynamic Custom Resources - create and delete resources ( #3742 )
2019-05-11 20:06:04 +08:00
686d4caefe
Updates to scheduling objects to support dynamic custom resources ( #4465 )
2019-04-27 18:45:23 -07:00
8dfc833a8b
Change all instances of JobID to DriverID. ( #4431 )
2019-04-22 16:28:09 -07:00
0f42f87ebc
Updating zero capacity resource semantics ( #4555 )
2019-04-12 16:53:57 -07:00
fe07a5b4b1
Add delete_creating_tasks option for internal.free() ( #4588 )
...
* add delete creating task objects.
* format code style
* Fix lint
* add tests add address comments.
* Refine test
* Refine java test
* Fix CI
* Refine
* Fix lint
* Fix CI
2019-04-12 13:38:31 +08:00
c2349cf12d
Remove local/global_scheduler from code and doc. ( #4549 )
2019-04-03 17:05:09 -07:00
b9ea821d16
Use strongly typed IDs in C++. ( #4185 )
...
* Use strongly typed IDs for C++.
* Avoid heap allocation in cython.
* Fix JNI part
* Fix rebase conflict
* Refine
* Remove type check from __init__
* Remove unused constructor declarations.
2019-03-07 21:43:01 +08:00
62055cc01c
Cleanup depulicated code of Cython ID types ( #4162 )
2019-02-26 16:19:12 +08:00
49dc85e54b
Fix wrong ID type in prepare_checkpoint ( #4124 )
...
* Fix wrong ID type in prepare_checkpoint
* fix
* fix eq
2019-02-25 11:53:09 -08:00
bcd5af78c7
Lint Cython files ( #4097 )
2019-02-20 22:29:25 -08:00
3549cd8195
Add the Delete function in GCS ( #4081 )
...
* Add the Delete function in GCS
* Unify BatchDelete and Delete
* Fix comment
* Lint
* Refine according to comments
* Unify test.
* Address comment
* C++ lint
* Update ray_redis_module.cc
2019-02-21 13:33:37 +08:00
7574757391
Fix crash for Java task's task.argument() in state. ( #4063 )
2019-02-19 12:46:07 +08:00
Eric Liang