28b1f7710c
[Core] Error info pubsub (Remove ray.errors API) ( #9665 )
2020-08-04 14:04:29 +08:00
9f969260e8
[core] Fix Ray service startup when logging redirection is disabled. ( #9547 )
2020-07-23 11:26:24 -05:00
d49dadf891
Change Python's ObjectID to ObjectRef ( #9353 )
2020-07-10 17:49:04 +08:00
46962f5db1
[Core] Log monitor multidriver ( #8953 )
2020-06-25 11:05:53 -07:00
c152730e4a
[Core] Log output from different jobs to different drivers. ( #8885 )
...
* .
* .
* Correct now
* No interactivity errors
* format
* Filtering
* lint
* .
* No more filtering
* Removed interactivity
* .
* .
* .
* .
* .
* .
* Redirection works
* formatting
* something broken?
* .
* Works
* formatting
* redirect output
* formatting
* formatting
* Fix file descriptor leakage
* format
* .
* .
* .
* .
* .
* Refactor
* .
* Only run on job switch
* .
* cleanup
* .
* ...
* Review
* .
* .
* .
* .
* whoops
* .
* Should fix bug
* .
* .
* addressed comments
* formatting
* formatting
* Fix typo
* .
* .
* .
* .
Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-33.us-west-2.compute.internal >
2020-06-23 18:45:32 -07:00
8958728139
Windows bug fixes ( #7740 )
2020-03-30 20:39:23 -05:00
fc23f79f82
Windows process issues ( #7739 )
2020-03-29 12:48:32 -07:00
ee8c9ff732
Remove six and cloudpickle from setup.py. ( #7694 )
2020-03-23 11:42:05 -07:00
a0700e2f86
Change /tmp to platform-specific temporary directory ( #7529 )
2020-03-16 18:10:14 -07:00
4d42664b2a
Use prctl(PR_SET_PDEATHSIG) on Linux instead of reaper ( #7150 )
2020-03-03 11:45:42 -06:00
0d3687a10d
No warning for docker memory > system memory ( #7151 )
2020-02-13 15:21:44 -08:00
72c31e3e19
Ray nodes should respect docker limits ( #7039 )
2020-02-10 11:08:38 -08:00
0826f95e1c
Including psutil & setproctitle ( #7031 )
2020-02-05 14:16:58 -08:00
52c33b53f7
[minor][core] fix gpu ids for SLURM ( #7014 )
...
* fix gpu ids
* fix
2020-02-02 16:09:22 -08:00
92525f35d1
Remove raylet client from Python worker ( #6018 )
2020-01-31 18:23:01 -08:00
c480d1d1e4
Treat static methods as class methods instead of instance methods in actors ( #6756 )
...
* Treat static methods as class methods rather than instance methods
* Add tests for static methods in actors
* Revert formatting changes
* Readd future imports
* Restructure static method check
* Documentation enhancements
* Fix linting issues
2020-01-15 19:38:41 -06:00
60d4d5e1aa
Remove future imports ( #6724 )
...
* Remove all __future__ imports from RLlib.
* Remove (object) again from tf_run_builder.py::TFRunBuilder.
* Fix 2xLINT warnings.
* Fix broken appo_policy import (must be appo_tf_policy)
* Remove future imports from all other ray files (not just RLlib).
* Remove future imports from all other ray files (not just RLlib).
* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).
* Add two empty lines before Schedule class.
* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
859dbad155
Fix estimate_available_memory() in utils.py ( #6302 )
2020-01-08 15:22:47 +08:00
5b40408678
[tune] Remove py2.7-specific code ( #6665 )
...
* Remove backwards compatability py2.7 code.
* Use exists_ok=True in ray
* nit
* nit
Co-authored-by: Richard Liaw <rliaw@berkeley.edu >
2020-01-03 01:03:13 -08:00
3db8faab0d
[tune] fix log dir race condition ( #6420 )
2019-12-10 21:00:19 -08:00
4edae7ea2b
Speed up task submissions a bit ( #5992 )
2019-10-25 00:10:37 -07:00
07c4c6367a
[core worker] Python core worker object interface ( #5272 )
2019-09-12 23:07:46 -07:00
dfd2a45f69
Simplify symlinking and don't print warnings ( #5615 )
2019-09-02 13:31:10 -07:00
0cc0abf857
Create session_latest symlink for Ray sessions ( #5580 )
2019-08-31 22:14:54 -07:00
e2e30ca507
Ray, Tune, and RLlib support for memory, object_store_memory options ( #5226 )
2019-08-21 23:01:10 -07:00
f2293243cc
[ID Refactor] Shorten the length of JobID to 4 bytes ( #5110 )
...
* WIP
* Fix
* Add jobid test
* Fix
* Add python part
* Fix
* Fix tes
* Remove TODOs
* Fix C++ tests
* Lint
* Fix
* Fix exporting functions in multiple ray.init
* Fix java test
* Fix lint
* Fix linting
* Address comments.
* FIx
* Address and fix linting
* Refine and fix
* Fix
* address
* Address comments.
* Fix linting
* Fix
* Address
* Address comments.
* Address
* Address
* Fix
* Fix
* Fix
* Fix lint
* Fix
* Fix linting
* Address comments.
* Fix linting
* Address comments.
* Fix linting
* address comments.
* Fix
2019-07-11 14:25:16 +08:00
5aec750107
Add warning/error if object store memory exceeds available memory ( #4893 )
...
* exclude
* format
* add warning
* hatch
* reduce mem usage
* reduce object store mem
* set obj mem
2019-07-08 21:37:08 -07:00
62e4b591e3
[ID Refactor] Rename DriverID to JobID ( #5004 )
...
* WIP
WIP
WIP
Rename Driver -> Job
Fix complition
Fix
Rename in Java
In py
WIP
Fix
WIP
Fix
Fix test
Fix
Fix C++ linting
Fix
* Update java/runtime/src/main/java/org/ray/runtime/config/RayConfig.java
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Update src/ray/core_worker/core_worker.cc
Co-Authored-By: Stephanie Wang <swang@cs.berkeley.edu >
* Address comments
* Fix
* Fix CI
* Fix cpp linting
* Fix py lint
* FIx
* Address comments and fix
* Address comments
* Address
* Fix import_threading
2019-06-28 00:44:51 +08:00
0131353d42
[gRPC] Migrate gcs data structures to protobuf ( #5024 )
2019-06-25 14:31:19 -07:00
1a39fee9c6
Refactor ID Serial 1: Separate ObjectID and TaskID from UniqueID ( #4776 )
...
* Enable BaseId.
* Change TaskID and make python test pass
* Remove unnecessary functions and fix test failure and change TaskID to
16 bytes.
* Java code change draft
* Refine
* Lint
* Update java/api/src/main/java/org/ray/api/id/TaskId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/BaseId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Update java/api/src/main/java/org/ray/api/id/ObjectId.java
Co-Authored-By: Hao Chen <chenh1024@gmail.com >
* Address comment
* Lint
* Fix SINGLE_PROCESS
* Fix comments
* Refine code
* Refine test
* Resolve conflict
2019-05-22 14:46:30 +08:00
bd00735fe8
Fix tempfile issues ( #4605 )
2019-05-05 16:06:15 -07:00
c2349cf12d
Remove local/global_scheduler from code and doc. ( #4549 )
2019-04-03 17:05:09 -07:00
41b81af11b
Downgrade six to 1.0.0 ( #4180 )
2019-02-27 13:05:25 -08:00
21472b890a
Integrate "tempfile_service" into "ray.node.Node" ( #3953 )
2019-02-12 17:34:04 -08:00
ef527f84ab
Stream logs to driver by default. ( #3892 )
...
* Stream logs to driver by default.
* Fix from rebase
* Redirect raylet output independently of worker output.
* Fix.
* Create redis client with services.create_redis_client.
* Suppress Redis connection error at exit.
* Remove thread_safe_client from redis.
* Shutdown driver threads in ray.shutdown().
* Add warning for too many log messages.
* Only stop threads if worker is connected.
* Only stop threads if they exist.
* Remove unnecessary try/excepts.
* Fix
* Only add new logging handler once.
* Increase timeout.
* Fix tempfile test.
* Fix logging in cluster_utils.
* Revert "Increase timeout."
This reverts commit b3846b89040bcd8e583b2e18cb513cb040e71d95.
* Retry longer when connecting to plasma store from node manager and object manager.
* Close pubsub channels to avoid leaking file descriptors.
* Limit log monitor open files to 200.
* Increase plasma connect retries.
* Add comment.
2019-02-07 19:53:50 -08:00
9295ab8f60
Various Python code cleanups. ( #3837 )
2019-02-03 10:16:24 -08:00
d128636bab
Ray Logging Configuration ( #3691 )
...
* fix logging for autoscaler
* module logging
* try this for logging
* yapf
* fix
* Initial logging setup
* momery
* ok
* remove basicconfig
* catch
* remove package logging
* print
* fix
* try_fix
* fix 1
* revert rllib
* logging level
* flake8
* fix
* fix
* Remove vestigal TODO
2019-01-30 21:01:12 -08:00
48139cf861
Migrate Python C extension to Cython ( #3541 )
2019-01-24 09:17:14 -08:00
d2cf8561f2
Refactor code about ray.ObjectID. ( #3674 )
...
* Refactor code about ray.ObjectID.
* remove from_random and use nil_id instead of constructor
* remove id() in hash
* Lint and fix
* Change driver id to ObjectID
* Replace binary_to_hex(ObjectID.id()) to ObjectID.hex()
2019-01-13 01:47:29 -08:00
c4b058739b
Remove redundant error message ( #3761 )
2019-01-12 22:22:41 -08:00
cd80891ddb
Try to figure out the memory limit in a docker container. ( #3605 )
...
* Try to figure out the memory limit in a docker container.
* Update comment
* Fix
* Fix
2019-01-03 23:07:24 -08:00
4b23a34c93
Fix multi-thread problem of function manager and Jenkins test ( #3648 )
2019-01-03 17:05:13 +08:00
bb7ca3bae7
Upgrade flatbuffers version to 1.10.0. ( #3559 )
...
* Upgrade flatbuffers version to 1.10.0.
* Temporarily change ray.utils.decode for backwards compatibility.
2018-12-23 14:56:34 -08:00
84fae57ab5
Convert the raylet client (the code in local_scheduler_client.cc) to proper C++. ( #3511 )
...
* refactoring
* fix bugs
* create client class
* create client class for java; bug fix
* remove legacy code
* improve code by using std::string, std::unique_ptr rename private fields and removing legacy code
* rename class
* improve naming
* fix
* rename files
* fix names
* change name
* change return types
* make a mutex private field
* fix comments
* fix bugs
* lint
* bug fix
* bug fix
* move too short functions into the header file
* Loose crash conditions for some APIs.
* Apply suggestions from code review
Co-Authored-By: suquark <suquark@gmail.com >
* format
* update
* rename python APIs
* fix java
* more fixes
* change types of cpython interface
* more fixes
* improve error processing
* improve error processing for java wrapper
* lint
* fix java
* make fields const
* use pointers for [out] parameters
* fix java & error msg
* fix resource leak, etc.
2018-12-13 13:39:10 -08:00
d950e92f63
Allow multiple threads to call ray.get and ray.wait ( #3244 )
...
* Handle multiple threads calling ray.get
* Multithreaded ray.wait
* Pass in current task ID in java backend
* Add multithreaded actor to tests, add warning messages to worker for multithreaded ray.get
* Fix test
* Some cleanups
* Improve error message
* Add assertion
* Cleanup, throw error in HandleTaskUnblocked if task not actually blocked
* lint
* Fix python worker reset
* Fix references to reconstruct_objects
* Linting
* java lint
* Fix java
* Fix iterator
2018-11-07 22:39:28 -08:00
9868af4c7c
Use /tmp instead of /dev/shm for object store on Linux if /dev/shm is too small. ( #3149 )
...
* Use /tmp instead of /dev/shm for object store on Linux if /dev/shm is too small.
* Add logging statement and address comments.
* Fix
2018-10-28 20:09:06 -07:00
658c14282c
Remove legacy Ray code. ( #3121 )
...
* Remove legacy Ray code.
* Fix cmake and simplify monitor.
* Fix linting
* Updates
* Fix
* Implement some methods.
* Remove more plasma manager references.
* Fix
* Linting
* Fix
* Fix
* Make sure class IDs are strings.
* Some path fixes
* Fix
* Path fixes and update arrow
* Fixes.
* linting
* Fixes
* Java fixes
* Some java fixes
* TaskLanguage -> Language
* Minor
* Fix python test and remove unused method signature.
* Fix java tests
* Fix jenkins tests
* Remove commented out code.
2018-10-26 13:36:58 -07:00
9948e8c11b
Move function/actor exporting & loading code to function_manager.py ( #3003 )
...
Move function/actor exporting & loading code to function_manager.py to prepare the code change for function descriptor for python.
2018-10-03 16:21:04 -07:00
ea9d1cc887
Remove dependence on psutil. Add utility functions for getting system memory. ( #2892 )
2018-09-18 15:03:29 +08:00
bd64c940e9
Push error to driver when monitor raises an exception. ( #2834 )
2018-09-07 17:42:45 -07:00
kisuke95