174bef56d4
[Core] Publish gcs server failure to drivers. ( #11265 )
...
* Done.
* Fixed.
2020-10-08 08:59:31 -07:00
dc7c2a70b8
[Core] Report worker backlog in GCS heartbeat ( #11039 )
2020-10-05 22:00:44 -07:00
518876ef20
Remove test_autoscaler_yaml from windows tests ( #11197 )
...
Co-authored-by: Ameer Haj Ali <ameerhajali@ameers-mbp.lan >
Co-authored-by: Alex Wu <alex@anyscale.io >
Co-authored-by: Alex Wu <itswu.alex@gmail.com >
Co-authored-by: Eric Liang <ekhliang@gmail.com >
2020-10-05 17:07:00 -07:00
80cc161f3e
[Placement Group] Report placement group load through heartbeat. ( #11129 )
...
* In progress.
* Fix a minor issue.
* Removed unnecessary comments.
* Addressed code review.
* Fix build failure.
* remove stray logs.
* Move global state to a med size test to avoid windows CI breakage.
2020-10-04 16:47:22 -07:00
6b86d4d280
Automatically detect CPU, GPU, accelerator_type for AWS ( #11147 )
2020-10-02 21:16:43 -07:00
ee71fecd9e
[Test] Make test_serialization a medium size test. ( #11183 )
2020-10-02 17:53:46 -07:00
ada58abcd9
[Object spilling] Update object directory and reload spilled objects automatically ( #11021 )
...
* Fix pytest...
* Release objects that have been spilled
* GCS object table interface refactor
* Add spilled URL to object location info
* refactor to include spilled URL in notifications
* improve tests
* Add spilled URL to object directory results
* Remove force restore call
* Merge spilled URL and location
* fix
* CI
* build
* osx
* Fix multitenancy issues
* Skip windows tests
2020-10-02 15:52:42 -07:00
0d93b1de93
[autoscaler] rsync to node ( #11122 )
2020-10-01 01:42:49 -07:00
8f73014291
[autoscaler] Make it more clear NodeProvider and CommandRunner are private APIs ( #11086 )
2020-09-30 16:24:37 -07:00
f0dba6bd2b
Code cleanup about python3 asyncio compat ( #11134 )
...
* cleanup python3 compat and others
2020-09-30 14:22:25 -07:00
0dcfa9ed6c
Add light heartbeat flag in python and use it in load metrics ( #11032 )
2020-09-30 11:39:28 -07:00
3504391fd2
[Core] Multi-tenancy: enable multi-tenancy by default ( #10570 )
...
* Add new job in Travis to enable multi-tenancy
* fix
* Update .bazelrc
* Update .travis.yml
* fix test_job_gc_with_detached_actor
* fix test_multiple_downstream_tasks
* fix lint
* Enable multi-tenancy by default
* Kill idle workers in FIFO order
* Update test
* minor update
* Address comments
* fix some cases
* fix test_remote_cancel
* Address comments
* fix after merge
* remove kill
* fix worker_pool_test
* fix java test timeout
* fix test_two_custom_resources
* Add a delay when killing idle workers
* fix test_worker_failure
* fix test_worker_failed again
* fix DisconnectWorker
* update test_worker_failed
* Revert some python tests
* lint
* address comments
2020-09-29 23:54:53 -07:00
f54f7b23f6
[Object Spilling] Name IO workers properly. ( #11097 )
...
* Fix.
* Addressed code review.
2020-09-29 23:48:31 -07:00
f17ad5d060
[autoscaler] resource demand scheduler get_nodes_to_launch should return Dict not List ( #11118 )
2020-09-29 15:32:56 -07:00
1db83764bf
[GCS]Use new getting all available resources interface instead of pub-sub … ( #10914 )
...
* Use new all available resources getting interface instead of pub-sub in state.py
* add missing server handler and test cases, fix comments
* add fine grained test assert
* per comments
* involve new added function _available_resources_per_node
* change ClientID to NodeID
* fix compile
* fix client id and lint
* robust tests check
* robust tests
2020-09-29 09:41:10 -07:00
0d36e4c025
[autoscaler] Support min_workers for multi node type ( #11041 )
...
* prepare for head node
* move command runner interface outside _private
* remove space
* Eric
* flake
* min_workers in multi node type
* fixing edge cases
* eric not idle
* fix target_workers to consider min_workers of node types
* idle timeout
* minor
* minor fix
* test
* lint
* eric v2
* eric 3
* min_workers constraint before bin packing
* Update resource_demand_scheduler.py
* Revert "Update resource_demand_scheduler.py"
This reverts commit 818a63a2c86d8437b3ef21c5035d701c1d1127b5.
* reducing diff
Co-authored-by: Ameer Haj Ali <ameerhajali@ameers-mbp.lan >
Co-authored-by: Alex Wu <alex@anyscale.io >
Co-authored-by: Alex Wu <itswu.alex@gmail.com >
2020-09-28 22:02:01 -07:00
cef263a333
[Custom Metrics] Improve ray runtime context. ( #10981 )
...
* done.
* Addressed code review.
* Addressed code review.
2020-09-28 16:18:19 -07:00
872219940b
[GCS]Fix miss PollOwnerForActorOutOfScope after gcs restarts bug ( #11054 )
...
* fix_RemoveActorFromOwner_crash_bug
* fix review comment
* fix review comment
* rm unused ut
* add testcase
* fix review comment
* rm unused import
* fix code style
* fix ut bug
Co-authored-by: 灵洵 <fengbin.ffb@antfin.com >
2020-09-28 10:06:40 -07:00
27e1f513e3
[Log] make glog flush and RAY_LOG thread-safe ( #11002 )
...
* make glog flush and RAY_LOG thread-safe
* dump error log to console
* mapping all levels to destination
* hack glog for exporting message to stdout if no base name given
* patch lint
* use stdout logger by default
* add raylet std/err pytest checker
* add worker logs file check
* fix asan check
* loop in glog enums
* fix python lint
* lint for autoindent
* fix indent lint
* make raylet.err is not empty
2020-09-28 22:15:15 +08:00
1e39c40370
[Placement Group] Capture child tasks by default. ( #11025 )
...
* In progress.
* Finished up.
* Improve comment.
* Addressed code review.
* Fix test failure.
* Fix ci failures.
* Fix CI issues.
2020-09-27 19:33:00 -07:00
b5ecdd088b
Deflake object manager test ( #11052 )
2020-09-26 18:33:02 -07:00
29663d89f1
[Placement Group] Remove warning msg for placement groups. ( #11034 )
...
* Done.
* Addressed code review.
* Fixed typo.
* Addressed code review.
2020-09-25 20:53:42 -07:00
109481afd9
[Metric] custom metrics refinement ( #10861 )
...
* In progress
* In Progress.
* Addressed code review.
* Add unit tests.
* Add a simple doc.
* Fixed test failure.
* Fix all test failures from serve.
* Addressed code review.
2020-09-25 09:10:28 -07:00
609c1b8acd
Start moving ray internal files to _private module ( #10994 )
2020-09-24 22:46:35 -07:00
3b6fe72029
Make the command runner interface public ( #11022 )
2020-09-24 22:45:17 -07:00
a26394d184
[tests] Blacklist failing win32 tests ( #10935 )
2020-09-24 20:15:28 -07:00
5e6b887f2d
[Placement Group] Capture Child Task Part 1 ( #10968 )
...
* In progress.
* In progers.
* Done.
* Addressed code review.
* Increase timeout to make a test less flaky.
* Addressed code review.
* Addressed code review.
2020-09-24 09:02:03 -07:00
46a560e876
[cli][ray] ray start should error by default if there's already an instance running ( #10826 )
2020-09-24 10:29:59 -05:00
295782d411
[New Scheduler] Refactor cluster resource scheduler ( #10938 )
2020-09-23 15:46:31 -07:00
567009d5fd
[Autoscaler] Fix k8s command runner when command fails ( #10966 )
...
Co-authored-by: Allen Yin <allenyin@anyscale.io >
2020-09-23 13:25:49 -07:00
864d1d2b59
[Core] Multi-tenancy: Kill idle workers in FIFO order ( #10597 )
...
* Kill idle workers in FIFO order
* Update test
* minor update
* Address comments
* fix after merge
* fix worker_pool_test
2020-09-22 10:59:11 -07:00
9d205cda86
[Tests] Fix flaky test_failures by using semaphore instead of sleep ( #10942 )
...
* Use semaphore to avoid timing issues.
* Addressed code review.
2020-09-21 23:39:49 -07:00
9da7bdcc8e
Use master for links to docs in source ( #10866 )
2020-09-19 00:30:45 -07:00
6a227ae501
[autoscaler] Split autoscaler interface public private ( #10898 )
2020-09-18 18:16:23 -07:00
9a07c7b963
[1.0] Remove args from ray start ( #10659 )
2020-09-18 16:41:23 -07:00
e56f2b8586
[autoscaler] hotfix calculate_node_resources ( #10874 )
2020-09-18 13:39:00 -07:00
a159ae72b3
[autoscaler] Ensure run_init happens in all cases ( #10820 )
...
* taking a stab at the horror that is NodeUpdaterThread
* added tests
2020-09-17 11:23:56 -07:00
269e1f0b98
Fix push_error_to_driver_through_redis ( #10848 )
...
Co-authored-by: 刘宝 <po.lb@antfin.com >
2020-09-17 10:50:44 -07:00
8a77cf925a
[cli][ray] update ray cli message ( #10823 )
2020-09-17 09:26:55 -07:00
fe4c6ab778
[Core] Remove unused credis related code. ( #10849 )
...
* Done.
* Lint.
2020-09-16 23:34:54 -07:00
6f479d4697
[hotfix] CPU Detection ( #10821 )
2020-09-16 21:02:52 -07:00
54c616e9f3
[autoscaler] Enforce CommandRunnerInterface abstractions ( #10822 )
...
Co-authored-by: root <root@ip-172-31-28-155.us-west-2.compute.internal >
Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local >
2020-09-16 16:56:03 -07:00
eb891e6c94
[TEST]make retry counts ( #10463 )
...
* make retry counts
* lint
* use function
* move methond inside
2020-09-16 10:12:30 -07:00
4c03f7ca2f
[Core] Multi-tenancy: Reject worker registration if job has finished ( #10569 )
2020-09-14 14:49:31 +08:00
5bc2ba38fd
[docker] Detect CPUs in container correctly ( #10507 )
...
Co-authored-by: simon-mo <simon.mo@hey.com >
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
Co-authored-by: Alex Wu <itswu.alex@gmail.com >
2020-09-13 23:40:48 -07:00
660aee6311
[cli] make test failure less verbose + print ssh ( #10767 )
2020-09-13 23:37:10 -07:00
9795356ac0
[hotfix] Autoscaler's K8 support ( #10766 )
...
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-09-13 16:43:57 -07:00
00fefba72c
[autoscaler] Usability improvements in logging ( #10764 )
2020-09-13 13:54:44 -07:00
d0b73647b4
[Autoscaler] Unmanaged nodes ( #10513 )
2020-09-13 11:58:47 -07:00
4659e24495
[docker] Revert to rsync & cp instead of file mount for bootstrap config/key ( #10734 )
2020-09-11 16:56:26 -07:00
SangBin Cho