Commit Graph
37 Commits
Author SHA1 Message Date
Eric LiangandGitHub a26394d184 [tests] Blacklist failing win32 tests (#10935) 2020-09-24 20:15:28 -07:00
567009d5fd [Autoscaler] Fix k8s command runner when command fails (#10966)
Co-authored-by: Allen Yin <allenyin@anyscale.io>
2020-09-23 13:25:49 -07:00
Eric LiangandGitHub 6a227ae501 [autoscaler] Split autoscaler interface public private (#10898) 2020-09-18 18:16:23 -07:00
Ian RodneyandGitHub a159ae72b3 [autoscaler] Ensure run_init happens in all cases (#10820)
* taking a stab at the horror that is NodeUpdaterThread

* added tests
2020-09-17 11:23:56 -07:00
Alex WuandGitHub d0b73647b4 [Autoscaler] Unmanaged nodes (#10513) 2020-09-13 11:58:47 -07:00
Ian RodneyandGitHub 4659e24495 [docker] Revert to rsync & cp instead of file mount for bootstrap config/key (#10734) 2020-09-11 16:56:26 -07:00
Alex WuandGitHub 23bbe0f36a [Autoscaler] Reload config (#10450) 2020-09-01 14:37:04 -07:00
Ian RodneyandGitHub 283f4d1060 [docker] Use tmp paths for rsync and fix file_mounts on docker (#10368) 2020-09-01 13:14:35 -07:00
Eric LiangandGitHub 85a6876119 [autoscaler] Rename instance_type => node_type, TAG_RAY_INSTANCE_TYPE => TAG_RAY_USER_NODE_TYPE (#10207) 2020-08-20 12:27:11 -07:00
Eric LiangandGitHub 538cb802d5 [autoscaler] Refactor multi node type autoscaler config (#10190) 2020-08-19 20:46:00 -07:00
Alex WuandGitHub 0b5d5ec17d [Autoscaler] Pass custom resources to "ray start" multi instance autoscaling (#9986) 2020-08-17 22:34:07 -07:00
Alan GuoandGitHub 3506910c5d [autoscaler] Create worker_file_mounts config (#9762) 2020-07-31 14:33:27 -07:00
Alan GuoandGitHub 5831737287 Introduce file_mounts_sync_continuously cluster option (#9544)
* Separate out file_mounts contents hashing into its own separate hash

Add an option to continuously sync file_mounts from head node to worker nodes:
monitor.py will re-sync file mounts whenver contents change but will only run setup_commands if the config also changes

* add test and default value for file_mounts_sync_continuously

* format code

* Update comments

* Add param to skip setup commands when only file_mounts content changed during monitor.py's update tick

Fixed so setup commands run when ray up is run and file_mounts content changes

* Refactor so that runtime_hash retains previous behavior

runtime_hash is almost identical as before this PR. It is used to determine if setup_commands need to run
file_mounts_contents_hash is an additional hash of the file_mounts content that is used to detect when only file syncing has to occur.

Note: runtime_hash value will have changed from before the PR because we hash the hash of the contents of the file_mounts as a performance optimization

* fix issue with hashing a hash

* fix bug where trying to set contents hash when it wasn't generated

* Fix lint error

Fix bug in command_runner where check_output was no longer returning the output of the command

* clear out provider between tests to get rid of flakyness

* reduce chance of race condition from node_launcher launching a node in the middle of an autoscaler.update call
2020-07-28 00:02:08 -07:00
1e46d4e29f [Autoscaler] Making bootstrap config part of the node provider interface (#9443)
* supporting custom bootstrap config for external node providers

* bootstrap config

* renamed config to cluster_config

* lint

* remove 2 args from importer

* complete move of bootstrap to node_provider

* renamed provider_cls

* move imports outside functions

* lint

* Update python/ray/autoscaler/node_provider.py

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* final fixes

* keeping lines to reduce diff

* lint

* lamba config

* filling in -> adding for lint

Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local>
Co-authored-by: Eric Liang <ekhliang@gmail.com>
2020-07-16 09:54:20 -07:00
Eric LiangandGitHub 82a5ed3fb5 [autoscaler] Instance type assert fix for non-AWS node providers (#9223) 2020-07-01 17:54:20 -07:00
Eric LiangandGitHub 536795ef79 [autoscaler] Initial support for multiple worker types (#9096)
* wip

* fix

* update

* debug state

* fix

* update

* update

* fix test

* fix

* fix

* update

* fix

* types and docs

* update
2020-06-24 23:08:30 -07:00
Eric LiangandGitHub 0ff24ec8dc Add "ray status" debug tool for autoscaler. (#9091) 2020-06-24 18:22:03 -07:00
mehrdadnandGitHub 981f67bfb0 Fix more Windows issues (#9011)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-19 18:51:45 -07:00
Alan GuoandGitHub 8f9b465968 Extract out "fillout_defaults" from other config preparation actions (#8996) 2020-06-17 16:40:37 -07:00
Eric LiangandGitHub a24d117c68 [autoscaler] Refactor code in preparation for multi instance type support (#8632)
* wip refactor

* add util

* wip

* fix

* fix

* remove

* remove extraneous string type for sg
2020-06-03 12:53:55 -07:00
mehrdadnandGitHub ebf060d484 Make more tests run on Windows (#8446)
* Remove worker Wait() call due to SIGCHLD being ignored

* Port _pid_alive to Windows

* Show PID as well as TID in glog

* Update TensorFlow version for Python 3.8 on Windows

* Handle missing Pillow on Windows

* Work around dm-tree PermissionError on Windows

* Fix some lint errors on Windows with Python 3.8

* Simplify torch requirements

* Quiet git clean

* Handle finalizer issues

* Exit with the signal number

* Get rid of wget

* Fix some Windows compatibility issues with tests

Co-authored-by: Mehrdad <noreply@github.com>
2020-05-20 12:25:04 -07:00
3c91ff1f63 [autoscaler] Allowing users to provide extra configs for AWS (#7844)
* Allowing users to provide custom key names & security group inbound rules

* linting

* getting aws credentials passed in

* one more thing

* one more thing part 2

* formatting

* addressing comments

* update

* update

* update

* update

* update

* update

* remove tests

* rerun tests

Co-authored-by: Allen Yin <allenyin@anyscale.io>
2020-04-04 18:36:51 -07:00
49439611f1 [autoscaler] Replace cluster yaml validation with json schema v… (#7261)
* replace manual cluster yaml validation with json schema
- improved error message
- support for intellisense in VSCode (or other IDEs)
- run linting
- moved schema to ray/autoscaler
- fixed typo
- remove importlib dependency

* Update python/ray/autoscaler/autoscaler.py

* read

* restrict allowed properties

* added unit test for invalid yaml
added ray[test] package (remove pytest from default dependencies)

* updated autoscaler test to use ValidationError exception

* add missing dependency

* added pytest

* replace manual cluster yaml validation with json schema
- improved error message
- support for intellisense in VSCode (or other IDEs)
- run linting
- moved schema to ray/autoscaler
- fixed typo
- remove importlib dependency

* Update python/ray/autoscaler/autoscaler.py

* read

* restrict allowed properties

* added unit test for invalid yaml
added ray[test] package (remove pytest from default dependencies)

* updated autoscaler test to use ValidationError exception

* add missing dependency

* added pytest

* removed parameterized dependency
reverted ray[test] intro

* removed parameterized

* fix_tests

* format

Co-authored-by: Ubuntu <marcozo@mc-ray-jumpbox.chcbtljllnieveqhw3e4c1ducc.xx.internal.cloudapp.net>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-03-10 18:58:55 -07:00
SvenandEric Liang 60d4d5e1aa Remove future imports (#6724)
* Remove all __future__ imports from RLlib.

* Remove (object) again from tf_run_builder.py::TFRunBuilder.

* Fix 2xLINT warnings.

* Fix broken appo_policy import (must be appo_tf_policy)

* Remove future imports from all other ray files (not just RLlib).

* Remove future imports from all other ray files (not just RLlib).

* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).

* Add two empty lines before Schedule class.

* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
Robert NishiharaandPhilipp Moritz 39a3459886 Remove (object) from class declarations. (#6658) 2020-01-02 17:42:13 -08:00
Eric LiangandGitHub 53641f1f74 Move more unit tests to bazel (#6250)
* move more unit tests to bazel

* move to avoid conflict

* fix lint

* fix deps

* seprate

* fix failing tests

* show tests

* ignore mismatch

* try combining bazel runs

* build lint

* remove tests from install

* fix test utils

* better config

* split up

* exclusive

* fix verbosity

* fix tests class

* cleanup

* remove flaky

* fix metrics test

* Update .travis.yml

* no retry flaky

* split up actor

* split basic test

* split up trial runner test

* split stress

* fix basic test

* fix tests

* switch to pytest runner for main

* make microbench not fail

* move load code to py3

* test is no longer package

* bazel to end
2019-11-24 11:43:34 -08:00
Eric LiangandGitHub 493364d3bd [autoscaler] Add unit tests for stopped node caching, fix flaky tests (#5793) 2019-09-27 22:36:09 -07:00
Edward OakesandGitHub 86610a30c9 [flaky test] Fix flaky checkpointing tests (#5791)
* Fix flaky checkpointing tests

* Fix checkpoint test logic

* Fix exception matching

* timeout exception

* Fix import

* fix build
2019-09-27 11:03:07 -07:00
Eric LiangandGitHub 3ed18d0b59 Fix edge case in autoscaler with poor bin packing (#5702)
* fix edge case

* fix for general case
2019-09-13 11:46:10 -07:00
Eric LiangandGitHub 2fdefe19b7 Take into account queue length in autoscaling (#5684) 2019-09-11 11:31:35 -07:00
Eric LiangandGitHub d20696300e Fix autoscaler format string for memory (#5542)
* add format string

* fix cast
2019-08-26 23:25:11 -07:00
Daniel EdgecumbeandRichard Liaw 06fec63c87 [autoscaler] Add a 'request_cores' function for manual autoscaling (#4754) 2019-07-26 17:14:45 -07:00
Daniel EdgecumbeandRobert Nishihara 3e1adafbce [autoscaler] Add an aggressive_autoscaling flag (#4285) 2019-04-13 18:44:32 -07:00
justinwyangandRobert Nishihara e88e706fcc Enforce quoting style in Travis. (#4589) 2019-04-11 14:24:26 -07:00
Kristian HartikainenandYuhong Guo ed02bf11f7 [autoscaler] Lint code that we forgot to lint in #4537 (#4584)
* Lint code that we forgot to lint in previous PR

* Revert setup command merge

* Lint

* Revert "Revert setup command merge"

This reverts commit 55e1cdb1f256ea51ef66a38730d8f7865f1f5ad1.

* Fix testReportsConfigFailures test

* Minor syntax tweaks

* Lint
2019-04-10 17:01:36 +08:00
Kristian HartikainenandEric Liang 524e69a82d [autoscaler] Change the get behavior of node providers' _get_node (#4132)
* Change the get behavior of GCPNodeProvider._get_node

* Add lock around the GCPNodeProvider._get_node call

* rename nodes

* lint

* Update GCPNodeProvider._get_node to match aws implementation

* assert

* log

* log highest heartbeats

* rename

* bringup to connected

* prune heartbeat times

* fix bringup
2019-02-24 18:43:35 -08:00
William MaandRobert Nishihara c7a4c74f55 Moving tests from test/ to python/ray/tests/ (#3950) 2019-02-21 11:09:08 -08:00