Siyuan (Ryans) Zhuang and GitHub
46cf433f0e
[Core] Remove Arrow dependencies ( #13157 )
...
* remove arrow ubsan
* remove arrow build depend
* remove arrow buffer
2021-01-04 11:19:09 -08:00
Gabriele Oliaro and GitHub
b4d688b4a6
Enabling the cancellation of non-actor tasks in a worker's queue ( #12117 )
...
* wrote code to enable cancellation of queued non-actor tasks
* minor changes
* bug fixes
* added comments
* rev1
* linting
* making ActorSchedulingQueue::CancelTaskIfFound raise a fatal error
* bug fix
* added two unit tests
* linting
* iterating through pending_normal_tasks starting from end
* fixup! iterating through pending_normal_tasks starting from end
* fixup! fixup! iterating through pending_normal_tasks starting from end
* post merge fixes
* added debugging instructions, pulled Accept() out of guarded loop
* removed debugging instructions, linting
2021-01-04 09:52:29 -08:00
c2bff64699
[Core] Locality-aware leasing: Milestone 1 - Owned refs, pinned location ( #12817 )
...
* Locality-aware leasing for owned refs (pinned locations).
* LessorPicker --> LeasePolicy.
* Consolidate GetBestNodeIdForTask and GetBestNodeIdForObjects.
* Update comments.
* Turn on locality-aware leasing feature flag by default.
* Move local fallback logic to LeasePolicy, move feature flag check to CoreWorker constructor, add local-only lease policy.
* Add lease policy consulting assertions to the direct task submitter tests.
* Add lease policy tests.
* LocalityLeasePolicy --> LocalityAwareLeasePolicy.
* Add missing const declarations.
Co-authored-by: SangBin Cho <rkooo567@gmail.com >
* Add RAY_CHECK for raylet address nullptr when creating lease client.
* Make the fact that LocalLeasePolicy always returns the local node more explicit.
* Flatten GetLocalityData conditionals to make it more readable.
* Add ReferenceCounter::GetLocalityData() unit test.
* Add data-intensive microbenchmarks for single-node perf testing.
* Add data-intensive microbenchmarks for simulated cluster perf testing.
* Remove redundant comment.
* Remove data-intensive benchmarks.
* Add locality-aware leasing Python test.
* Formatting changes in ray_perf.py.
Co-authored-by: SangBin Cho <rkooo567@gmail.com >
2021-01-04 09:49:08 -08:00
fangfengbin and GitHub
25f9f0d781
[GCS] Move resource usage info to gcs resource manager ( #13059 )
2020-12-25 15:17:45 +08:00
Siyuan (Ryans) Zhuang and GitHub
cf9952a028
[Core] Remote outdated external store ( #13080 )
...
* remove outdated external store
2020-12-24 17:30:06 -08:00
Siyuan (Ryans) Zhuang and GitHub
bf7f6a7de3
[Core] Remove cuda support in plasma store ( #13070 )
...
* remove cuda support in plasma store
2020-12-24 13:24:56 -08:00
4461f9980a
Refactor TaskDependencyManager, allow passing bundles of objects to ObjectManager ( #13006 )
...
* New dependency manager
* Switch raylet to new DependencyManager
* PullManager accepts bundles
* Cleanup, remove old task dependency manager
* x
* PullManager unit tests
* lint
* Unit tests
* Rename
* lint
* test
* Update src/ray/raylet/dependency_manager.cc
Co-authored-by: SangBin Cho <rkooo567@gmail.com >
* Update src/ray/raylet/dependency_manager.cc
Co-authored-by: SangBin Cho <rkooo567@gmail.com >
* x
* lint
Co-authored-by: SangBin Cho <rkooo567@gmail.com >
2020-12-23 18:36:00 -08:00
Stephanie Wang and GitHub
d95c8b8a41
[core][new scheduler] Move tasks from ready to dispatch to waiting on argument eviction ( #13048 )
...
* Add index for tasks to dispatch
* Task dependency manager interface
* Unsubscribe dependencies and tests
* NodeManager
* Revert "Add index for tasks to dispatch"
This reverts commit c6ccb9aa306e00f80d34b991055e4e83872595ea.
* tmp
* Move back to waiting if args not ready
* update
2020-12-23 09:33:43 -08:00
DK.Pino and GitHub
6e19facc7f
[GCS] Delete redis gcs client and redis_xxx_accessor ( #12996 )
2020-12-23 20:31:46 +08:00
fangfengbin and GitHub
646c4201ac
[GCS]Decouple gcs resource manager and gcs node manager ( #13012 )
2020-12-23 11:25:01 +08:00
fyrestone and GitHub
62a5832007
[Dashboard] Add GET /logical/actors API ( #12913 )
2020-12-23 11:14:23 +08:00
Alex Wu and GitHub
ea8d782be1
[core] Pull Manager exponential backoff ( #13024 )
2020-12-21 19:17:51 -08:00
Eric Liang and GitHub
8068041006
Don't release resources during plasma fetch ( #13025 )
2020-12-21 18:32:40 -08:00
Eric Liang and GitHub
03a5b90ed6
Revert "Revert "Increase the number of unique bits for actors to avoi… ( #12990 )
2020-12-21 15:16:42 -08:00
Kai Yang and GitHub
5a6801dde7
[Core] Remove delete_creating_tasks ( #12962 )
2020-12-22 00:01:27 +08:00
fangfengbin and GitHub
85a4435ba0
[GCS]Fix redis store client AsyncPutWithIndex unordered bug ( #13002 )
2020-12-21 20:02:50 +08:00
Barak Michener and GitHub
c576f0b073
[ray_client] Implement a gRPC streaming logs API for the client ( #13001 )
2020-12-20 19:35:34 -08:00
fangfengbin and GitHub
4caa6c6d78
[GCS]GCS resource manager remove cluster_resources_ ( #12972 )
2020-12-21 11:00:25 +08:00
Barak Michener and GitHub
e715ade2d1
Support retrieval of named actor handles ( #13000 )
...
Change-Id: I05d31c9c67943d2a0230782cbdaa98341584cbc7
2020-12-20 16:34:50 -08:00
Barak Michener and GitHub
80f6dd16b2
[ray_client] Implement optional arguments to ray.remote() and f.options() ( #12985 )
2020-12-20 15:43:48 -08:00
Barak Michener and GitHub
7ab9164f1b
[ray_client] Integrate with test_basic, test_basic_2 and test_actor ( #12964 )
2020-12-20 14:54:18 -08:00
3fab93b61b
Fix scheduling_resources comment errors ( #12991 )
...
* Fix scheduling_resources comment error
* add part code
Co-authored-by: 灵洵 <fengbin.ffb@antgroup.com >
2020-12-20 20:20:07 +08:00
Eric Liang and GitHub
64c97d25d3
Enable by default new scheduler ( #12735 )
2020-12-19 13:22:24 -08:00
Eric Liang and GitHub
5d987f5988
Revert "Increase the number of unique bits for actors to avoid handle collisions ( #12894 )" ( #12988 )
...
This reverts commit 3e492a79ec
2020-12-18 23:51:44 -08:00
dHannasch and GitHub
a092433bc8
[core] Use the ConnectWithoutRetries error message ( #12732 )
2020-12-18 22:34:34 -08:00
SangBin Cho and GitHub
9d939e6674
[Object Spilling] Implement level triggered logic to make streaming shuffle work + additional cleanup ( #12773 )
2020-12-18 19:31:14 -08:00
Alex Wu and GitHub
404161a3ff
[Autoscaler/Core] Remove autoscaler spam ( #12952 )
2020-12-18 18:22:45 -08:00
ac5ea2c13d
[Java] Fix output parsing in RunManager ( #12968 )
...
* Fix output parsing in RunManager
* change log level
Co-authored-by: 灵洵 <fengbin.ffb@antgroup.com >
2020-12-18 18:22:12 -08:00
Eric Liang and GitHub
6ece291f35
Clean up block/unblock handling of resources in new scheduler ( #12963 )
2020-12-18 16:00:54 -08:00
Eric Liang and GitHub
3e492a79ec
Increase the number of unique bits for actors to avoid handle collisions ( #12894 )
2020-12-18 15:59:03 -08:00
Eric Liang and GitHub
92812f2e8a
Implement resource deadlock detection for new scheduler ( #12961 )
2020-12-18 12:17:54 -08:00
Barak Michener and GitHub
5cfa1934e4
[ray_client]: Implement object retain/release and Data Streaming API ( #12818 )
2020-12-18 11:47:38 -08:00
a442cd17e0
[GCS]Optimize gcs client reconnection ( #12878 )
...
* [GCS]Optimize gcs client reconnection
* fix review comment
* fix review comment
* add part code
Co-authored-by: 灵洵 <fengbin.ffb@antgroup.com >
2020-12-17 21:57:37 -08:00
cfefd7c70e
Test PingPort ( #12954 )
...
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-12-17 21:15:42 -08:00
DK.Pino and GitHub
6404f1e609
[Placement Group][New scheduler] New scheduler pg implementation ( #12910 )
2020-12-18 11:56:45 +08:00
Tao Wang and GitHub
17152c84a7
[Tiny]Print raylet info after register ( #12566 )
2020-12-18 11:22:13 +08:00
dHannasch and GitHub
d747071dd9
Test shard_context on already-created boost::asio::io_service. ( #12917 )
2020-12-17 14:26:30 -08:00
e6cb4f4bd7
[Core] Add log of address and port ( #12908 )
...
Co-authored-by: Allen Yin <allenyin@anyscale.io >
2020-12-17 00:25:29 -08:00
Yi Cheng and GitHub
40032541dc
[core] Introduce fetch_local to ray.wait ( #12526 )
2020-12-16 23:44:28 -08:00
Tao Wang and GitHub
12231ec2a6
Optimize heartbeat manager initialization ( #12911 )
2020-12-17 14:24:23 +08:00
SangBin Cho and GitHub
057687e534
[New Scheduler] Fix test_failure.py by supporting infeasible tasks ( #12738 )
...
* Fix the first issue.
* ip
* In Progress.
* In progress.
* done.
* Remove unnecessary logs.
* Addressed code review + fix some test failures.
* Try fixing issues.
* Fix issues.
* Fix test issues.
* Fix issues.
* done.
2020-12-16 21:27:50 -08:00
Alex Wu and GitHub
8b783ecafa
Fix pull manager retry ( #12907 )
2020-12-16 14:18:43 -08:00
91878d18b5
[PlacementGroup]Fix placement group wait api disorder bug ( #12827 )
...
* [PlacementGroup]Fix placment group wait api disorder bug
* fix review comment
* fix review comment
* fix review comment
* fix review comments
* increase num_heartbeats_timeout
Co-authored-by: 灵洵 <fengbin.ffb@antgroup.com >
2020-12-16 18:45:53 +08:00
Eric Liang and GitHub
7ff314a5df
[New scheduler] Also unsubscribe get dependencies on unblock
2020-12-15 20:29:44 -08:00
Alex Wu and GitHub
0031723ace
[New scheduler] Object spilling ( #12857 )
2020-12-15 11:05:38 -08:00
Edward Oakes and GitHub
261b2f9053
Check for raylet PID as ppid in dashboard agent fate-sharing ( #12867 )
2020-12-15 12:13:11 -06:00
Max Fitton and GitHub
e077bc4206
[Release] Bump master to 1.2.0 for 1.1.0 release ( #12856 )
2020-12-15 09:40:26 -08:00
Simon Mo and GitHub
b291dd4486
[Metrics] Call GetMeasureDoubleByName to prevent override ( #12860 )
2020-12-15 09:39:39 -08:00
43b9259d40
[GCS]GCS resource manager support scheduling resource ( #12780 )
...
* add part code
* add part code
* fix review comments
* rebase master
* add part code
* add part code
* fix review comments
* add part code
* fix code style
* fix ut bug
* fix ut bug
* fix review comments
* fix review comment
Co-authored-by: 灵洵 <fengbin.ffb@antgroup.com >
2020-12-15 10:27:55 +08:00
Tao Wang and GitHub
ac53e2f857
[GCS]Tell dead nodes to commit suicide ( #12792 )
...
* [GCS]Tell dead nodes to commit suicide
* fix comment, add ut
2020-12-14 11:42:00 -08:00
