wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-09-10 12:38:43 +08:00
Code Issues Packages Projects Releases Wiki Activity
6,812 Commits 36 Branches 42 Tags
python3.9
Commit Graph
100 Commits
This Branch
This Branch
All Branches
Author SHA1 Message Date
SangBin Cho 9d939e6674 [Object Spilling] Implement level triggered logic to make streaming shuffle work + additional cleanup (#12773) 2020-12-18 19:31:14 -08:00
SangBin Cho 057687e534 [New Scheduler] Fix test_failure.py by supporting infeasible tasks (#12738) 
* Fix the first issue.

* ip

* In Progress.

* In progress.

* done.

* Remove unnecessary logs.

* Addressed code review + fix some test failures.

* Try fixing issues.

* Fix issues.

* Fix test issues.

* Fix issues.

* done.
 2020-12-16 21:27:50 -08:00
SangBin Cho de7848231c [Doc] Fix placement group doc (#12875) 2020-12-15 10:36:51 -08:00
SangBin Cho 69b0bc2132 [Logging] Use file handle temporalily (#12839) 2020-12-14 11:42:44 -08:00
SangBin Cho 162f361dab [Logging] Fix log monitor issue (#12588) 
* Try fixing issues.

* Verficiation.
 2020-12-07 22:01:18 -08:00
SangBin Cho b1f2b142d5 [Core] Ensure global state is connected when exception hook is called from the driver. (#12655) 2020-12-07 18:28:32 -08:00
SangBin Cho 040cf2c13b [Doc] Placement group doc small update (#12594) 
* Modify doc  that wasn't supposed to be merged.

* Addressed coder eview.
 2020-12-07 13:58:27 -08:00
SangBin Cho 3ee4612696 [Release] Fix cluster.yaml (#12589) 
* Fix cluster.yaml

* Updated to use manylinux2014
 2020-12-07 13:52:30 -08:00
SangBin Cho 0138c2dbb4 [Metrics] Remove redundant unit specification. (#12595) 2020-12-04 00:06:21 -08:00
SangBin Cho 981df65b91 [Doc] Improve the placement group document (#12507) 
* Improve the placement group document.

* Fix grammar.

* Addressed code review.
 2020-12-01 13:15:30 -08:00
SangBin Cho 0e892908f7 [Object Spilling] Delete spilled objects when references are gone out of scope. (#12341) 2020-12-01 13:10:39 -08:00
SangBin Choandchaokunyang f6f3cc9af1 [Core]Remove checkpoint table (#12235) 
* Delete an actor entry from node manager.

* Remove checkpoint table

* remote checkpoint interface

* remove checkpoint interface

* fix ExitActorTest

Co-authored-by: chaokunyang <shawn.ck.yang@gmail.com>
 2020-12-01 08:58:36 -08:00
SangBin Cho 8223a33bff [Logging] Log rotation on all components (#12101) 
* In Progress.

* Done.

* Fix the issue.

* Add wait for condition because logs are not written right away now.

* debug string.

* lint.

* Fix flaky test.

* Fix issues.

* Fix test.

* lint.
 2020-11-30 19:03:55 -08:00
SangBin Cho 3964defbe1 [Logging] Fix tensorflow logging issue. (#12225) 
* in progress.

* ip

* In Progress

* done.

* fix lint.

* Addressed code review

* Addressed code review.
 2020-11-29 22:16:52 -08:00
SangBin Cho 91d54ef621 [Core] Remove actor arg from executor to allow users to specify actor… (#12239) 
* [Core] Remove actor arg from executor to allow users to specify actor arg in their Actor.remote.

* Addressed code review.
 2020-11-29 22:15:48 -08:00
SangBin Cho 753cda2f28 [Dashboard] Delete old dashboard (#12144) 
* Delete old dashboard from repo.

* Delete old dashboard from repo. 2
 2020-11-25 11:31:02 -08:00
SangBin Cho 2e4e285ef0 [Object Spilling] Fusion small objects (#12087) 2020-11-25 10:13:32 -08:00
SangBin Cho eb89c567e2 [Metrics] Skip prom file test on Windows. (#12234) 2020-11-22 00:32:18 -08:00
SangBin Cho 5fb410cfbf [Dashboard] New dashboard view data doesn't exist. (#12129) 
* Fix.

* Fix the issue.
 2020-11-19 11:04:59 -08:00
SangBin Cho 7d67af6c2a [Metrics] Add stats to measure process startup time + scheduling stats. (#12100) 
* Add new stats.

* Fix issues.
 2020-11-19 11:04:26 -08:00
SangBin Cho f56d7c1a76 [Logging] Remove per worker job log file / support worker log rotation (#11927) 
* In progress.

* MVP done.

* In Progress.

* Remove unnecessay code.

* Fix some issues.

* Fix test failures.

* Addressed code review + fix object spilling test failure.
 2020-11-16 11:29:43 -08:00
SangBin Cho f6f9b15299 . (#11998) 2020-11-12 21:33:00 -08:00
SangBin Cho f80d812799 [Object Spilling] Introduce SpillWorker & RestoreWorker Pool to avoid IO worker deadlock. (#11885) 2020-11-11 18:20:14 -08:00
SangBin Cho 7b8bd15702 [Stalebot] Fix issues. (#11930) 2020-11-11 00:28:02 -08:00
SangBin Cho e0ecf5d79d Revert "[GCS]Open light heartbeat by default (#11689)" (#11861) 
This reverts commit 612ddb2dd1.
 2020-11-06 14:34:59 -08:00
SangBin Cho 3cd1d7f44a [Metrics] Implement basic metrics changes (#11769) 
* Implement basic metrics changes

* Addressed code review.

* Fix build issue.

* Fix build issue.
 2020-11-05 11:07:05 -08:00
SangBin Cho 049df70289 [OSS] Introduce Stale bot (#11790) 
* first iteration.

* Add a newline at the end of yaml

* Addressed code review.

* Addressed review.
 2020-11-05 11:02:37 -08:00
SangBin Cho 71c5089854 [Object Spilling] Initial Iteration of S3 adapter. (#11379) 
* Finished the first iteration.

* Removed unnecessary code.

* Smartopen impl.

* Make sure tests passed.

* Addressed code review.

* Addressed code review.

* Fix issues.

* Fix issues.
 2020-10-30 14:47:07 -07:00
SangBin Cho 6e2a1eac36 [Placement Group] Placement group automatic cleanup. (#11546) 
* In progress. Done with all placement group manager code.

* It is working with job.

* Finished detached actor implementation.

* Fix minor issue.

* In progress.

* Addressed code review.

* Addressed code review.

* Addressed code reivew.

* Fix a build error.
 2020-10-30 10:55:43 -07:00
SangBin Cho 666fcde8ca [Placement group] Input validation (#11152) 
* Add a basic input validation.

* Addressed code review.
 2020-10-14 13:56:41 -07:00
SangBin Cho b1481c6acf Revert "[PlacementGroup]Add node manager test framework (#11174)" (#11398) 
This reverts commit 241e765d3a.
 2020-10-14 11:09:20 -07:00
SangBin Cho c107eea551 [Core] Do not report stats when worker is already dead. (#11167) 
* Fix.

* Addressed code reivew.

* Done.
 2020-10-12 11:57:04 -07:00
SangBin Cho 56f69543d0 Try to deflake test_failure (#11293) 2020-10-12 12:03:36 -05:00
SangBin Cho 9dd4561d1b [Placement Group] Fix stress tests to pass when actors are scheduled. (#11151) 
* Fix stress tests to pass when actors are created.

* Addressed code review.
 2020-10-09 21:52:26 -07:00
SangBin Cho 6cb00208f7 [Placement Group] Export bundle reservation check method only once. (#11153) 
* Export bundle reservation check method only once.

* Addressed code review.
 2020-10-08 16:08:28 -07:00
SangBin Cho 174bef56d4 [Core] Publish gcs server failure to drivers. (#11265) 
* Done.

* Fixed.
 2020-10-08 08:59:31 -07:00
SangBin Cho 37fa86f9a0 [Placement Group] Fix placement group bugs that happen when rescheduling. (#11263) 
* Fix placement group bugs while autoscaling.

* Addressed code review.
 2020-10-08 08:58:59 -07:00
SangBin Cho 80cc161f3e [Placement Group] Report placement group load through heartbeat. (#11129) 
* In progress.

* Fix a minor issue.

* Removed unnecessary comments.

* Addressed code review.

* Fix build failure.

* remove stray logs.

* Move global state to a med size test to avoid windows CI breakage.
 2020-10-04 16:47:22 -07:00
SangBin Cho 6974cea0cd [Core] Use optional return instead of nullptr for the GetNode method. 2020-10-02 20:54:26 -07:00
SangBin Cho ee71fecd9e [Test] Make test_serialization a medium size test. (#11183) 2020-10-02 17:53:46 -07:00
SangBin Cho e44828d928 [Test] Make the test size bigger for tune to fix timeout. (#11138) 2020-09-30 11:35:31 -07:00
SangBin Cho 474af36591 [Core] Fix random bugs found. (#11128) 2020-09-30 00:11:57 -07:00
SangBin Cho f54f7b23f6 [Object Spilling] Name IO workers properly. (#11097) 
* Fix.

* Addressed code review.
 2020-09-29 23:48:31 -07:00
SangBin Cho 0a6164ab15 [Core] Improve logging messages. (#11082) 2020-09-28 21:07:45 -07:00
SangBin Cho cef263a333 [Custom Metrics] Improve ray runtime context. (#10981) 
* done.

* Addressed code review.

* Addressed code review.
 2020-09-28 16:18:19 -07:00
SangBin Cho 1e39c40370 [Placement Group] Capture child tasks by default. (#11025) 
* In progress.

* Finished up.

* Improve comment.

* Addressed code review.

* Fix test failure.

* Fix ci failures.

* Fix CI issues.
 2020-09-27 19:33:00 -07:00
SangBin Cho 29663d89f1 [Placement Group] Remove warning msg for placement groups. (#11034) 
* Done.

* Addressed code review.

* Fixed typo.

* Addressed code review.
 2020-09-25 20:53:42 -07:00
SangBin Cho 8abe13023f [Metric] Fix issue 10634 (#10940) 
* Fix.

* Revert "Fix."

This reverts commit 52c9c1ee646b551a4dd2b639c78be67683db2b1c.

* ADdressed code review.

* Addressed code review.
 2020-09-25 09:11:05 -07:00
SangBin Cho 109481afd9 [Metric] custom metrics refinement (#10861) 
* In progress

* In Progress.

* Addressed code review.

* Add unit tests.

* Add a simple doc.

* Fixed test failure.

* Fix all test failures from serve.

* Addressed code review.
 2020-09-25 09:10:28 -07:00
SangBin Cho 8c241d5f1d [Core] Use node ip address properly in ray.init (#10829) 
* Fix.

* Addressed code review.

* Addressed code review.
 2020-09-24 11:44:52 -07:00
SangBin Cho 5e6b887f2d [Placement Group] Capture Child Task Part 1 (#10968) 
* In progress.

* In progers.

* Done.

* Addressed code review.

* Increase timeout to make a test less flaky.

* Addressed code review.

* Addressed code review.
 2020-09-24 09:02:03 -07:00
SangBin Cho cebab8886e [Placement group] Refine doc (#10922) 2020-09-23 16:26:32 -07:00
SangBin Cho 7931b6ce2e Fix placement group bug failing in release test (#10944) 2020-09-23 12:37:28 -07:00
SangBin Cho 390107b6cb [Core] Allow to pass node ip address to gcs server. (#10946) 
* Allow to pass node ip address to gcs server.

* Fix.

* Addressed code review.

* Fixed an error.

* Addressed code review.
 2020-09-23 01:52:26 -07:00
SangBin Cho e3b4850224 [Placement group] Release test (#10924) 
* Done.

* Lint.

* Addressed code review.
 2020-09-22 00:49:04 -07:00
SangBin Cho 9d205cda86 [Tests] Fix flaky test_failures by using semaphore instead of sleep (#10942) 
* Use semaphore to avoid timing issues.

* Addressed code review.
 2020-09-21 23:39:49 -07:00
SangBin Cho bc74a10748 [Core] Fix Flaky GCS actor manager test (#10600) 
* Try.

* Fix the issue.

* Fix.
 2020-09-17 16:10:57 -07:00
SangBin Cho fedbdd5dc6 [Doc] Broken doc build fix. (#10865) 2020-09-17 13:27:30 -07:00
SangBin Cho fe4c6ab778 [Core] Remove unused credis related code. (#10849) 
* Done.

* Lint.
 2020-09-16 23:34:54 -07:00
SangBin ChoandRichard Liaw 1fdb7ef6c3 [docs] Placement group documentation (#10555) 
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
 2020-09-16 16:07:55 -07:00
SangBin ChoandRichard Liaw d7d4e1c87b [Doc] Document options method (#10830) 
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
 2020-09-16 14:23:24 -07:00
SangBin Cho 517e164fb7 [Core] Update the object manager pulling objects error message to warning. (#10657) 
* Update the message to expose less implementation details and make the severity WARNING.

* Fix formatting.
 2020-09-11 15:53:04 -07:00
SangBin Cho b7040f1310 Revert "[Streaming] fix streaming ci (#9675)" (#10656) 
This reverts commit 3645a05644.
 2020-09-08 19:07:21 -07:00
SangBin Cho 30fff71d96 087release update (#10651) 
* [BRING BACK TO MASTER] Release test result for microbenchmark.

* [BRING BACK TO MASTER] Update result and fix sanity check bugs.
 2020-09-08 15:45:06 -07:00
SangBin Cho dcb9e03fde [Placement Group] Atomic Creation using 2 phase protocol part 2. (#10599) 
* In progress.

* In Progress

* Basic done.

* Fix build issues.

* Addressed code review.

* Change the confusing test name.

* Fix comments.

* Addressed code review.
 2020-09-08 13:11:11 -07:00
SangBin Cho cb919c5e5c Revert "[Streaming] Fault Tolerance Implementation (#10008)" (#10582) 
This reverts commit 1b1466748f.
 2020-09-04 13:21:18 -07:00
SangBin Cho 2a7f56e429 [Placement group] Fix Logging issues. (#10557) 2020-09-03 23:55:10 -07:00
SangBin Cho dc7fe1a4c5 [Placement Group] Atomic Placement Group Part 1, Basic Structure. (#10482) 
* Write a test.

* Basic structure done.

* Reduce flakiness of tests.

* Addressed code review.

* Skipping tests because it is flaky for now.

* Fix linting issues.

* Increase sleep time to see lint messages.

* Lint issue fixed.
 2020-09-02 18:14:46 -07:00
SangBin Cho a0c7907d88 [Placement Group] Leasing context refactoring part 2 (#10413) 
* In progress.

* Refactoring done, but still failing tests.

* Fix issues.

* Addressed code review.

* Addressed code review.
 2020-08-31 15:54:34 -07:00
SangBin Cho 3e5cac80d8 [Tests] Fix Broken GCS restart test. (#10417) 2020-08-30 00:44:34 -07:00
SangBin Cho c8b14fd7e9 [Tests] Enable large test (#10391) 
* Enable large tests.

* Lint.

* Fix issue.

* Skip all tests.
 2020-08-30 00:44:05 -07:00
SangBin Cho 68c2dcd12b Fix. (#10390) 2020-08-28 08:22:23 -07:00
SangBin Cho d206fbbc99 [Placement group] Scheduler map refactoring part 1. (#10381) 
* In Progress

* done.

* Address code review.
 2020-08-28 00:57:09 -07:00
SangBin Cho 7b29eb7949 [Build] Try parallel Python builds. (#10291) 
* Trial 1.

* Parallelize even more.
 2020-08-28 00:06:52 -07:00
SangBin Cho f35339b5ff [Dashboard] Change default ip address for the dashboard to ipv4 (#10287) 
* Done.

* Add todo.

* Addressed code review.

* Fix issue.

* Fix test failure.

* Fix a test.
 2020-08-27 14:43:10 -07:00
SangBin Cho 17f465d5c1 [Core] Improve raylet failure error msg (#10345) 
* Improve error message.

* Lint.

* Addressed code review.
 2020-08-27 12:53:18 -07:00
SangBin Cho f846b26165 [Doc] Add ports configuration & Move dashboard to the original place (#10281) 
* Done.

* Address code review.

* Addressed code review.
 2020-08-27 12:00:16 -07:00
SangBin Cho 7ea4bcc1ab Add a basic rule to contributors / PR template. (#10277) 
* Add a basic rule to contributors / PR template.

* Fix.

* Addressed code reivew.
 2020-08-24 20:15:06 -07:00
SangBin Cho 3b3ca96a4e [Placement Group] Wait (#10259) 
* Initial progress done.

* Fix wrong test.

* Improve tests.

* Update code.

* Addressed code review and merge conflict.

* Addressed code review.
 2020-08-24 20:14:48 -07:00
SangBin Cho 1f54acd274 [Tech Debt] Use f-string for python/ray/*.py (#10268) 
* In progress.

* Done with critical path.

* Modified cluster_utils.py and log_monitor.py

* Addressed code review.
 2020-08-23 22:01:31 -07:00
SangBin Cho 92664249e8 Partially Use f string (#10218) 
* flynt. trial 1.

* Trial 1.

* Addressed code review.
 2020-08-20 18:21:16 -07:00
SangBin Cho 224933b5e4 [Placement Group] Remove API part 2 (#10215) 
* Initial progress done.

* Fix mistake.

* Addressed code review.

* Fix cpp build issue.

* Addressed code review.
 2020-08-20 09:50:13 -07:00
SangBin Cho 263df6163c [Placement Group] Placement group remove api part 1 (#10063) 
* Added basic rpc calls.

* fix issues.

* Fix the gcs server not getting request issue.

* In Progress.

* Basic logic done. Tests are required.

* In progress.

* In progress in refactoring context.

* Revert "In progress in refactoring context."

This reverts commit 38236256cf1306c60dd203e75d45ceb4509c8106.

* Working now.

* Python test works.

* Lint.

* Addressed code review.

* Addressed code review.

* Lint.

* Added unit tests.

* Done, but one of unit tests fail

* Addressed code review.

* Addressed the last code review.

* Fix the wrong test case.
 2020-08-18 12:44:00 -07:00
SangBin Cho 8cedcdf2df [Tests] Fix test output (#10162) 
* Trial 1.

* Fix.

* Revert "Fix."

This reverts commit 26ad970f753d581f340857be30054d6954df8255.

* Revert "Trial 1."

This reverts commit 63f7aca5162bb40f2d5e28fb9647598cbde7ad41.

* Another fix try.

* Last trial.

* Remove unnecessary comment.

* Small fix.

* Use better units.

* Lint.
 2020-08-17 21:24:20 -07:00
SangBin Cho 053188dfbe [Placement Group] Support Placement Group state table. (#10090) 
* Done.

* Addressed code review.

* Linting.

* Fix lint.

* Fix lint.

* Fix a test.

* Lint.

* Add a lint sleep to test.

* Fix the lint issue.

* Fixed doc build error.
 2020-08-17 09:24:50 -07:00
SangBin Cho 16f6ee4914 Small Update (#10107) 2020-08-13 19:39:55 -07:00
SangBin Cho 55fe7f65a5 [Tests] Make test_output debugging easy (#10091) 
* Fix.

* Fix.
 2020-08-13 18:45:26 -07:00
SangBin Cho 8b689224a5 [Tests] Make test_multi_driver light. (#10086) 2020-08-13 10:00:42 -07:00
SangBin Cho 86b1db3f11 [Stats] Make metrics report time configurable (#10036) 
* Done.

* Lint.

* Address code review.

* Address code review.

* Remove wrong commit.

* Fix a test error.
 2020-08-13 00:30:24 -07:00
SangBin Cho 2cb79632e4 Revert "[Core] Add cached memory to available memory (#10020)" (#10064) 
This reverts commit 71d2bde458.
 2020-08-12 11:24:16 -05:00
SangBin Cho 946ae74817 [GCS Actor Management] Race condition around creating -> created phase. (#10035) 
* Fix the issue.

* Address a code review.
 2020-08-11 12:31:27 -07:00
SangBin Cho 10baecb8c2 Try to fix. (#10005) 2020-08-10 10:05:43 -07:00
SangBin Cho eb6b10221e Increase the num of trials to reduce the probability of failing sample_test (#10007) 2020-08-10 10:05:33 -07:00
SangBin Cho 39088ab6f2 [Stats] Metrics Export User Interface Part 2 (Prometheus Service Discovery) (#9970) 
* In progress.

* In Progress.

* Finish the working version.

* Write a documentation.

* Addressed code review.

* Fix lint error.

* Lint.

* Addressed code review. Make test less flaky.

* Use a random port for ray start.

* Modify doc.

* Make write atomic.
 2020-08-07 21:59:24 -07:00
SangBin Cho 326a470bc2 [Test] Reduce the wait for condition timeout. (#9971) 2020-08-07 11:44:53 -07:00
SangBin Cho 22934619df [Release] Fix release tests (#9976) 
* In Progress.

* Fix torch version.

* Fix asan run.sh

* Addressed code review.
 2020-08-07 10:06:13 -07:00
SangBin Cho 44826878ff [Core] Remove Legacy Raylet Code (#9936) 
* Remove a flag and some methods in node manager including HandleDisconnectedActor, ResubmitTask, and HandleTaskReconstruction

* Make actor creator always required + remove raylet transport

* Remove actor reporter + remove FinishAssignedActorCreationTask

* Remove actor tasks.

* Remove finishactortask and switched it to finishactorcreation task

* Remove reconstruction policy.

* Remove lineage cache.

* Formatting.

* Remove actor frontier code.

* Removed build error.

* Revert "Remove reconstruction policy."

This reverts commit 9d25c9bced4da5fbcac5d484d51013345f16513b.

* Recover HandleReconstruction to mark expired objects as failed.
 2020-08-06 16:37:50 -07:00
SangBin Cho ec2f1a225e [Stats] Metrics Export User Interface Part 1 (#9913) 
* Metrics export port expose done.

* Support exposing metrics port + metrics agent service discovery through ray.nodes()

* Formatting.

* Added a doc.

* Linting.

* Change the location of metrics agent port.

* Addressed code review.

* Addressed code review.
 2020-08-06 16:16:29 -07:00
SangBin Cho 68899e2f8e [GCS Actor Management] Fix race condition for DEPENDENCIES_UNREADY states. (#9883) 
* Fix issues.

* Address code review.

* Addressed code review 2.

* Fix formatting.

* Addressed code review 3/

* Addressed code review.
 2020-08-05 12:22:12 -07:00
SangBin Cho 685182923c [Core] Fix detached actor local mode when gcs actor management is on. (#9839) 
* Fix local mode detached actor.

* Revert changes.
 2020-08-05 09:04:24 -07:00