54189bca5a
[autoscaler] Create Docker Command Runner ( #8806 )
...
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-06-05 17:29:27 -07:00
2c485a2598
[autoscaler] Command runner interactivity ( #7198 )
2020-06-05 17:08:38 -07:00
d452932740
[autoscaler] Improve Logtimer log messages ( #8753 )
2020-06-04 18:07:27 -07:00
d966d98729
cleanup to support provider's custom ssh command runner ( #8720 )
...
* cleanup to support provider's custom ssh command runner
* clean up
* trailing white spaces fix
* k8s signature fix
Co-authored-by: Ameer Haj Ali <ameerhajali@Ameers-MacBook-Pro.local >
2020-06-04 13:27:17 -07:00
09f89ff49d
[autoscaler] Improve SSH Command Failure Logging ( #8751 )
2020-06-04 12:38:20 -07:00
a24d117c68
[autoscaler] Refactor code in preparation for multi instance type support ( #8632 )
...
* wip refactor
* add util
* wip
* fix
* fix
* remove
* remove extraneous string type for sg
2020-06-03 12:53:55 -07:00
474bbc28bf
Warn if Autoscaling-config flag not set. ( #8677 )
2020-06-03 12:21:07 -07:00
30ed20405a
[autoscaler] Support creating services in k8s backend ( #8659 )
2020-05-29 15:19:21 -05:00
76450c8d47
[autoscaler] Honor separate head and worker node subnet IDs ( #8374 )
2020-05-28 18:16:46 -07:00
13231ba63b
Rename redis-port to port and add default ( #8406 )
2020-05-18 13:25:34 -05:00
ec9357b486
[autoscaler] Fix filesystem permission race conditions ( #8327 )
2020-05-05 17:22:03 -07:00
5bc6e32c0a
[autoscaler] latest_dlami update ( #8178 )
2020-04-26 00:25:46 -07:00
0dc01d8c1e
[autoscaler] Azure versioning ( #8168 )
2020-04-24 17:03:55 -07:00
3c91ff1f63
[autoscaler] Allowing users to provide extra configs for AWS ( #7844 )
...
* Allowing users to provide custom key names & security group inbound rules
* linting
* getting aws credentials passed in
* one more thing
* one more thing part 2
* formatting
* addressing comments
* update
* update
* update
* update
* update
* update
* remove tests
* rerun tests
Co-authored-by: Allen Yin <allenyin@anyscale.io >
2020-04-04 18:36:51 -07:00
b853df7a3b
[autoscaler] Switch to ARM for Azure deployment ( #7717 )
...
* switch to ARM templates for config and VMs
* switch to ARM templates for config and VMs
* auto-formatting
* addressed Scotts comment
* added missing imports
* fixed gpu templates
fixed wheel reference
* added missing reference
* cleanup wording and yamls
* Update doc/source/autoscaling.rst
Co-Authored-By: Scott Graham <5720537+gramhagen@users.noreply.github.com >
Co-authored-by: Ubuntu <marcozo@marcozodev2.zqvgrdyupqrudayw1il1agipig.jx.internal.cloudapp.net >
Co-authored-by: Scott Graham <5720537+gramhagen@users.noreply.github.com >
2020-04-03 15:51:56 -07:00
37e4d29f87
[autoscaler] Adding Azure Support ( #7080 )
...
* adding directory and node_provider entry for azure autoscaler
* adding initial cut at azure autoscaler functionality, needs testing and node_provider methods need updating
* adding todos and switching to auth file for service principal authentication
* adding role / scope to service principal
* resolving issues with app credentials
* adding retry for setting service principal role
* typo and adding retry to nic creation
* adding nsg to config, moving nic/public ip to node provider, cleanup node_provider, leaving in NodeProvider stub for testing
* linting
* updating cleanup and fixing bugs
* adding directory and node_provider entry for azure autoscaler
* adding initial cut at azure autoscaler functionality, needs testing and node_provider methods need updating
* adding todos and switching to auth file for service principal authentication
* adding role / scope to service principal
* resolving issues with app credentials
* adding retry for setting service principal role
* typo and adding retry to nic creation
* adding nsg to config, moving nic/public ip to node provider, cleanup node_provider, leaving in NodeProvider stub for testing
* linting
* updating cleanup and fixing bugs
* minor fixes
* first working version :)
* added tag support
* added msi identity intermediate
* enable MSI through user managed identity
* updated schema
* extend yaml schema
remove service principal code
add re-use of managed user identity
* fix rg_id
* fix logging
* replace manual cluster yaml validation with json schema
- improved error message
- support for intellisense in VSCode (or other IDEs)
* run linting
* updating yaml configs and formatting
* updating yaml configs and formatting
* typo in example config
* pulling default config from example-full
* resetting min, init worker prop
* adding docs for azure autoscaler and fixing status
* add azure to docs, fix config for spot instances, update azure provider to avoid caching issues during deployment
* fix for default subscription in azure node provider
* vm dev image build
* minor change
* keeping example-full.yaml in autoscaler/azure, updating azure example config
* linting azure config
* extending retries on azure config
* lint
* support for internal ips, fix to azure docs, and new azure gpu example config
* linting
* Update python/ray/autoscaler/azure/node_provider.py
Co-Authored-By: Richard Liaw <rliaw@berkeley.edu >
* revert_this
* remove_schema
* updating configs and removing ssh keygen, tweak azure node provider terminate
* minor tweaks
Co-authored-by: Markus Cozowicz <marcozo@microsoft.com >
Co-authored-by: Ubuntu <marcozo@mc-ray-jumpbox.chcbtljllnieveqhw3e4c1ducc.xx.internal.cloudapp.net >
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-03-15 14:48:27 -07:00
49439611f1
[autoscaler] Replace cluster yaml validation with json schema v… ( #7261 )
...
* replace manual cluster yaml validation with json schema
- improved error message
- support for intellisense in VSCode (or other IDEs)
- run linting
- moved schema to ray/autoscaler
- fixed typo
- remove importlib dependency
* Update python/ray/autoscaler/autoscaler.py
* read
* restrict allowed properties
* added unit test for invalid yaml
added ray[test] package (remove pytest from default dependencies)
* updated autoscaler test to use ValidationError exception
* add missing dependency
* added pytest
* replace manual cluster yaml validation with json schema
- improved error message
- support for intellisense in VSCode (or other IDEs)
- run linting
- moved schema to ray/autoscaler
- fixed typo
- remove importlib dependency
* Update python/ray/autoscaler/autoscaler.py
* read
* restrict allowed properties
* added unit test for invalid yaml
added ray[test] package (remove pytest from default dependencies)
* updated autoscaler test to use ValidationError exception
* add missing dependency
* added pytest
* removed parameterized dependency
reverted ray[test] intro
* removed parameterized
* fix_tests
* format
Co-authored-by: Ubuntu <marcozo@mc-ray-jumpbox.chcbtljllnieveqhw3e4c1ducc.xx.internal.cloudapp.net >
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-03-10 18:58:55 -07:00
d192ef0611
[raysgd] Cleanup User API ( #7384 )
...
* Init fp16
* fp16 and schedulers
* scheduler linking and fp16
* to fp16
* loss scaling and documentation
* more documentation
* add tests, refactor config
* moredocs
* more docs
* fix logo, add test mode, add fp16 flag
* fix tests
* fix scheduler
* fix apex
* improve safety
* fix tests
* fix tests
* remove pin memory default
* rm
* fix
* Update doc/examples/doc_code/raysgd_torch_signatures.py
* fix
* migrate changes from other PR
* ok thanks
* pass
* signatures
* lint'
* Update python/ray/experimental/sgd/pytorch/utils.py
* Apply suggestions from code review
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com >
* should address most comments
* comments
* fix this ci
* first_pass
* add overrides
* override
* fixing up operators
* format
* sgd
* constants
* rm
* revert
* save
* failures
* fixes
* trainer
* run test
* operator
* code
* op
* ok done
* operator
* sgd test fixes
* ok
* trainer
* format
* Apply suggestions from code review
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com >
* Update doc/source/raysgd/raysgd_pytorch.rst
* docstring
* dcgan
* doc
* commits
* nit
* testing
* revert
* Start renaming pytorch to torch
* Rename PyTorchTrainer to TorchTrainer
* Rename PyTorch runners to Torch runners
* Finish renaming API
* Rename to torch in tests
* Finish renaming docs + tests
* Run format + fix DeprecationWarning
* fix
* move tests up
* benchmarks
* rename
* remove some args
* better metrics output
* fix up the benchmark
* benchmark-yaml
* horovod-benchmark
* benchmarks
* Remove benchmark code for cleanups
* makedatacreator
* relax
* metrics
* autosetsampler
* profile
* movements
* OK
* smoothen
* fix
* nitdocs
* loss
* comments
* fix
* fix
* runner_tests
* codes
* example
* fix_test
* fix
* tests
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com >
Co-authored-by: Maksim Smolin <maximsmol@gmail.com >
2020-03-10 08:41:42 -07:00
b74eb5fce6
Capture output for commands run by the autoscaler ( #7381 )
2020-03-03 10:19:21 -08:00
734629b4ea
Ssh command format ( #7176 )
2020-02-17 14:15:42 -08:00
52d9189d5d
[autoscaler] port-forward for attach + redis_port ( #7145 )
...
* port-forward
* fixport
* force redis port in init mode
* test
* Update python/ray/tests/test_ray_init.py
2020-02-14 15:17:00 -08:00
4139e02f01
[autoscaler] Add `--all-nodes` option to rsync-up ( #7065 )
...
* Add option to sync workers to rsync-up
* Format
* Rename --sync-workers to --all-nodes
2020-02-10 16:27:59 -08:00
3f99be8dad
Add 'ray dashboard' command ( #6959 )
2020-02-10 12:55:21 -08:00
37053443b4
Restore set omp ( #7051 )
2020-02-04 15:02:23 -08:00
cc43c9c1a2
Increase limit for autoscaler keys ( #7007 )
2020-02-01 22:29:40 -08:00
8b4b49662b
Force OMP_NUM_THREADS=1 if unset ( #6998 )
...
* force omp
* update
* set
* workers
* link
2020-02-01 11:46:11 -08:00
d9a2294298
Ssh identities only ( #6931 )
2020-01-27 17:01:21 -06:00
e0078a0d78
[autoscaler][minor] default -> latest_dlami ( #6922 )
...
* config
* latest
* Update python/ray/autoscaler/aws/config.py
2020-01-27 14:34:07 -08:00
e516c50745
[autoscaler]: Kill workers if the monitor raises an exception ( #3977 )
...
Co-authored-by: CJosephides <cjosephides@gmail.com >
2020-01-23 14:12:52 -06:00
2b0e93586f
[autoscaler] Auto-replace "DEFAULT" with most recent DLAMI ( #6848 )
...
* try_this
* fix
* actual fix
* default
2020-01-21 13:54:04 -08:00
60d4d5e1aa
Remove future imports ( #6724 )
...
* Remove all __future__ imports from RLlib.
* Remove (object) again from tf_run_builder.py::TFRunBuilder.
* Fix 2xLINT warnings.
* Fix broken appo_policy import (must be appo_tf_policy)
* Remove future imports from all other ray files (not just RLlib).
* Remove future imports from all other ray files (not just RLlib).
* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).
* Add two empty lines before Schedule class.
* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
39a3459886
Remove (object) from class declarations. ( #6658 )
2020-01-02 17:42:13 -08:00
d2c6457832
Remove public facing references to --redis-address. ( #6631 )
2019-12-31 13:21:53 -08:00
735f282494
Use 0.9.0.dev0 as the version tag ( #6630 )
2019-12-30 10:14:07 -08:00
8724e5ffd5
Start WebUI by default. ( #6493 )
2019-12-27 13:49:07 -08:00
1a1324d2a2
Bump version from 0.8.0.dev6 -> 0.9.0.dev ( #6508 )
2019-12-16 23:57:42 -08:00
b7d23405fe
[Dashboard] Change default port from 8080 to 8265 ( #6503 )
...
* [Dashboard] Change default port from 8080 to 8265
* Revise order of imports in pip install setup command
2019-12-16 14:25:23 -08:00
f63b64310a
Bump version to 0.8.0.dev7 ( #6303 )
2019-12-05 18:33:54 -08:00
dff6017272
Fix "failed to create head node" issue ( #6304 )
...
* Fix failed to create head node issue
* comments
2019-12-02 15:22:00 -08:00
fc655acfee
Fix linting on master branch ( #6174 )
2019-11-16 10:02:58 -08:00
e8cce3fdd4
[autoscaler]: automatically pull new docker image ( #6111 )
...
* Docker: automatically pull new image
* Fix missing value in schema
* Address review comments
2019-11-15 21:26:28 -08:00
385783fcec
Ray on YARN + Skein Documentation ( #6119 )
2019-11-14 15:06:05 -08:00
c75ada9e04
[Autoscaler][K8s] Enforce memory limit in k8s yaml ( #6138 )
...
* Enforce memory limit in k8s yaml
* Update python/ray/autoscaler/kubernetes/example-full.yaml
Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com >
* Line wrap
2019-11-11 14:06:34 -08:00
01aee8d970
[autoscaler] Retry creating EC2 instances in new AZ ( #6129 )
2019-11-09 19:44:27 -08:00
29eee7f970
Forward multiple ports for autoscaler ( #5893 )
2019-10-18 16:50:46 -07:00
6c11b534c8
[Autoscaler] Update AWS Deep Learning AMI to version 24.3 ( #5932 )
2019-10-16 16:50:54 -07:00
69d5c1b53a
remove evil redirects ( #5919 )
2019-10-14 19:41:04 -07:00
20c0cdee4f
[autoscaler] Worker-Head termination + Better Scale-up message ( #5909 )
2019-10-14 10:37:50 -07:00
abbfe7392f
Bump dev version to 0.8.0.dev6 ( #5906 )
2019-10-14 11:36:13 +01:00
0f24509c30
[autoscaler] uptime redirect fix ( #5907 )
...
* small change
* comment
2019-10-13 23:25:15 +01:00
Ian Rodney