Commit Graph
2670 Commits
Author SHA1 Message Date
Max FittonandMax Fitton 735cc40162 The required resources field is not always present, leading the logical view to crash when it is not present. (#9959)
Co-authored-by: Max Fitton <max@semprehealth.com>
2020-08-06 17:07:30 -07:00
Max FittonandMax Fitton ccd383bcd5 [Dashboard] Fix Bug in Machine View when Unsorted with Multiple Machines (#9938)
* Patch issue where when the Machine view was unsorted and grouped, it would crash.

* lint

Co-authored-by: Max Fitton <max@semprehealth.com>
2020-08-06 11:04:04 -07:00
Richard Liaw 9c19b14b1e [cli] Maintain "old-style" for abort (#9943) 2020-08-06 11:03:52 -07:00
Alex Wu b0ad10c936 [Core] test_advanced_3.py::test_logging_to_driver (round 2) (#9916) 2020-08-06 11:03:39 -07:00
Max FittonandMax Fitton c1c65619d1 [Dashboard] Update ActorState in dashboard to support new actor states (#9855)
* Update ActorState in dashboard to support new actor states

* Update dashboard documentation for new states

* Add missing state to doc

Co-authored-by: Max Fitton <max@semprehealth.com>
2020-08-06 11:02:17 -07:00
SangBin Cho aa6f0b7a6c [Core] Fix detached actor local mode when gcs actor management is on. (#9839)
* Fix local mode detached actor.

* Revert changes.
2020-08-06 11:02:09 -07:00
Amog KamsettyandKai Fricke 92e0718218 [Tune] Transformer blog example (#9789)
Co-authored-by: Kai Fricke <kai@anyscale.com>
2020-08-06 11:01:53 -07:00
Alex WuandUbuntu b58e4b5ee4 [Core] Fix test_logging_to_driver (#9829)
* Fixed

* .

* .

Co-authored-by: Ubuntu <ubuntu@ip-172-31-7-236.us-west-2.compute.internal>
2020-08-06 11:01:29 -07:00
Richard Liaw a96ddec358 [tune] Fix restoration for function API PBT (#9853) 2020-08-06 11:00:52 -07:00
Alex WuandEric Liang ea1ac15da0 [Core] Gpu type detection (#9695)
* .

* .

* .

* .

* .

* .

* .

* .

* Test cases

* detection only

* .

* Done?

* .

* .

* Done

* added test case

* .

* .

* .

* .

* .

* .

* Update python/ray/ray_constants.py

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* .

* .

Co-authored-by: Eric Liang <ekhliang@gmail.com>
2020-08-06 11:00:37 -07:00
Alan Guo f8f6f342f6 [autoscaler] Create worker_file_mounts config (#9762) 2020-08-06 11:00:30 -07:00
SangBin Cho 0a6847c5dc Bump up the veresion. 2020-07-31 10:09:54 -07:00
Richard Liaw a47121476f [tune] Remove accidentally added files (#9835) 2020-07-30 21:47:27 -07:00
mehrdadn a7b97b6f8a Add shellcheck support (#8574) 2020-07-30 18:39:28 -05:00
SangBin Cho 940617d092 Make test failure large. (#9822) 2020-07-30 13:11:51 -07:00
619e44e54a [tune] Added WandbLogger (#9725)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2020-07-30 13:09:03 -07:00
Barak Michener 68f3fec744 *: Centralize requirements.txt and unify dependency versions (#9759)
* python_test: fix cython_examples in doc/ and tests/

* update setup.py to parse the bazel version string better

* all: centralize all python deps into stackable requirements files in python/

* format

* Move cython test into the proper package

* Add cross-reference dependency comments for requirements and setup.py

* re-enable version pinning on CI, fix formatting

* fix up torchvision version

* fix case in shell
2020-07-30 11:22:56 -07:00
0c3b9ebeef [tune/sgd] Document func_trainable and add checkpoint context (#9739)
Co-authored-by: krfricke <krfricke@users.noreply.github.com>
Co-authored-by: Amog Kamsetty <amogkam@users.noreply.github.com>
2020-07-30 09:46:37 -07:00
SangBin Cho 826f14c824 [Stats] Fix harvestor threads + Fix flaky stats shutdown. (#9745) 2020-07-29 18:57:59 -05:00
Alex Wu 6e294dd90f [Core] Custom socket name (#9766)
* fix issues

* hot fixes

* test

* test

* socket name change only
2020-07-29 13:19:41 -07:00
Alex Wu e6696b2533 Fixed stderr logging (9765) 2020-07-29 13:19:04 -07:00
fangfengbin a484947742 Fix leased worker leak bug if lease worker requests that are still waiting to be scheduled when GCS restarts (#9719) 2020-07-29 14:16:03 +08:00
mehrdadnandMehrdad fb5280f21b Fix some Windows CI issues (#9708)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-28 18:10:23 -07:00
Alex Wu 21af0ceb0c Register function race (#9346) 2020-07-28 13:51:34 -07:00
SangBin Cho 7e3ba289dc [Stats] Basic Metrics Infrastructure (Metrics Agent + Prometheus Exporter) (#9607) 2020-07-28 10:28:01 -07:00
Ian Rodney b1c2983c97 Run _with_interactive in Docker (#9747) 2020-07-28 08:57:04 -07:00
Alan Guo 5831737287 Introduce file_mounts_sync_continuously cluster option (#9544)
* Separate out file_mounts contents hashing into its own separate hash

Add an option to continuously sync file_mounts from head node to worker nodes:
monitor.py will re-sync file mounts whenver contents change but will only run setup_commands if the config also changes

* add test and default value for file_mounts_sync_continuously

* format code

* Update comments

* Add param to skip setup commands when only file_mounts content changed during monitor.py's update tick

Fixed so setup commands run when ray up is run and file_mounts content changes

* Refactor so that runtime_hash retains previous behavior

runtime_hash is almost identical as before this PR. It is used to determine if setup_commands need to run
file_mounts_contents_hash is an additional hash of the file_mounts content that is used to detect when only file syncing has to occur.

Note: runtime_hash value will have changed from before the PR because we hash the hash of the contents of the file_mounts as a performance optimization

* fix issue with hashing a hash

* fix bug where trying to set contents hash when it wasn't generated

* Fix lint error

Fix bug in command_runner where check_output was no longer returning the output of the command

* clear out provider between tests to get rid of flakyness

* reduce chance of race condition from node_launcher launching a node in the middle of an autoscaler.update call
2020-07-28 00:02:08 -07:00
SangBin Cho 914cc96c91 Fix broken actor failure tests. (#9737) 2020-07-27 18:59:44 -07:00
Alisa 51e12ee97c Python api of placement group (#9243) 2020-07-27 14:57:05 -07:00
Robert Nishihara db0d6e8efa Make wait_for_condition raise exception when timing out. (#9710) 2020-07-26 22:56:32 -07:00
fyrestone 4d08ddbf24 [Dashboard] New dashboard skeleton (#9099) 2020-07-27 11:34:47 +08:00
Tao Wang 44ccca1acb Only update raylet map when autoscaler configured (#9435) 2020-07-27 11:23:06 +08:00
ZhuSenlin a269ae9bc4 [GCS] Fix actor task hang when its owner exits before local dependencies resolved (#8045) 2020-07-27 10:56:52 +08:00
Richard Liaw f3fdb5c5db [tune] distributed torch wrapper (#9550)
* changes

* add-working

* checkpoint

* ccleanu

* fix

* ok

* formatting

* ok

* tests

* some-good-stuff

* fix-torch

* ddp-torch

* torch-test

* sessions

* add-small-test

* fix

* remove

* gpu-working

* update-tests

* ok

* try-test

* formgat

* ok

* ok
2020-07-26 09:37:22 -07:00
fangfengbinand灵洵 ff1d7a6b36 [GCS]Open test_gcs_fault_tolerance testcase (#9677)
* enable test_gcs_fault_tolerance

* fix lint error

Co-authored-by: 灵洵 <fengbin.ffb@antfin.com>
2020-07-26 11:26:45 +08:00
Robert Nishihara a8efb214de Fix a few flaky tests (#9709)
Fix test_custom_resources, Remove test_pandas_parquet_serialization, Better error message for test_output.py, Potentially fix test_dynres::test_dynamic_res_creation_scheduler_consistency
2020-07-25 17:11:38 -07:00
SangBin Cho d49b19c24c [Stats] Improve Stats::Init & Add it to GCS server (#9563) 2020-07-25 10:42:08 -07:00
9f3570828a [tune] move jenkins tests to travis (#9609)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
Co-authored-by: Kai Fricke <kai@anyscale.com>
2020-07-24 21:22:54 -07:00
Maksim SmolinandRichard Liaw d364afbd31 [Autoscaler] CLI Logger docs (#9690)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-07-24 16:59:25 -07:00
mehrdadnandMehrdad 56d2cf6479 Shellcheck rewrites (#9597)
* Fix SC2001: See if you can use ${variable//search/replace} instead.

* Fix SC2010: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

* Fix SC2012: Use find instead of ls to better handle non-alphanumeric filenames.

* Fix SC2015: Note that A && B || C is not if-then-else. C may run when A is true.

* Fix SC2028: echo may not expand escape sequences. Use printf.

* Fix SC2034: variable appears unused. Verify use (or export if used externally).

* Fix SC2035: Use ./*glob* or -- *glob* so names with dashes won't become options.

* Fix SC2071: > is for string comparisons. Use -gt instead.

* Fix SC2154: variable is referenced but not assigned

* Fix SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

* Fix SC2188: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

* Fix SC2236: Use -n instead of ! -z.

* Fix SC2242: Can only exit with status 0-255. Other data should be written to stdout/stderr.

* Fix SC2086: Double quote to prevent globbing and word splitting.

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 17:24:19 -05:00
Simon Mo a078a21437 [Serve] Allow multiple HTTP servers. (#9523) 2020-07-24 12:41:20 -07:00
Stephanie Wang f2705e2c73 [core] Enable object reconstruction for retryable actor tasks (#9557)
* Test actor plasma reconstruction

* Allow resubmission of actor tasks

* doc

* Test for actor constructor

* Kill PID before removing node

* Kill pid before node
2020-07-23 21:15:12 -07:00
Robert Nishihara 06c3518aa1 Drop support for Python 3.5. (#9622)
* Drop support for Python 3.5.

* Update setup.py
2020-07-23 19:26:06 -07:00
krfrickeandKai Fricke ea4797bf38 [RaySGD] revised existing transformer example to work with transformers>=3.0 (#9661)
Co-authored-by: Kai Fricke <kai@anyscale.com>
2020-07-23 10:58:09 -07:00
Clark Zinzow 9f969260e8 [core] Fix Ray service startup when logging redirection is disabled. (#9547) 2020-07-23 11:26:24 -05:00
Maksim SmolinandRichard Liaw 908c0c630a [Autoscaler] Command Line Interface improvements (#9322)
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-07-22 12:21:44 -07:00
Max Fitton 456e012029 Revert "Dashboard next-version API support in backend (#9345)" (#9639)
This reverts commit fca1fb18f3.
2020-07-22 13:30:12 -05:00
mehrdadnandMehrdad 6346c70792 Fix bug in Bazel version check (#9626)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 20:58:46 -07:00
mehrdadnandMehrdad b14728d999 Shellcheck quoting (#9596)
* Fix SC2006: Use $(...) notation instead of legacy backticked `...`.

* Fix SC2016: Expressions don't expand in single quotes, use double quotes for that.

* Fix SC2046: Quote this to prevent word splitting.

* Fix SC2053: Quote the right-hand side of == in [[ ]] to prevent glob matching.

* Fix SC2068: Double quote array expansions to avoid re-splitting elements.

* Fix SC2086: Double quote to prevent globbing and word splitting.

* Fix SC2102: Ranges can only match single chars (mentioned due to duplicates).

* Fix SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

* Fix SC2145: Argument mixes string and array. Use * or separate argument.

* Fix SC2209: warning: Use var=$(command) to assign output (or quote to assign string).

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 21:56:41 -05:00
ZhuSenlin 382b314241 [GCS] fix the fault tolerance about gcs node manager (#9380) 2020-07-22 10:55:51 +08:00