Commit Graph
1541 Commits
Author SHA1 Message Date
Robert NishiharaandPhilipp Moritz 7792032ee3 Fix UI issue for non-json-serializable task arguments. (#1892)
* Fix UI issue for non-json-serializable task arguments.

* Simplify approach.
2018-04-15 13:54:42 -07:00
Robert NishiharaandPhilipp Moritz 3383553dc0 Remove unnecessary calls to .hex() for object IDs. (#1910) 2018-04-15 13:52:51 -07:00
Robert NishiharaandPhilipp Moritz 6f8b81d9e5 Allow multiple raylets to be started on a single machine. (#1904) 2018-04-15 13:51:19 -07:00
Stephanie WangandRobert Nishihara 4b655b0ff6 [xray] Turn on flushing to the GCS for the lineage cache (#1907) 2018-04-14 23:40:56 -07:00
Melih ElibolandPhilipp Moritz fcd30444a8 Single Big Object Parallel Transfer. (#1827)
* cache all object info from object added store notification.

* Adds parallel transfer for big objects.

* documentation and clean up.

* compare objects...

* merge buffer_state with chunk vec. Make separate buffer state for get and create.

* use references for Get. Allow partial failure of Create.

* single plasma client.

* changes based on review.

* update documentation and add parameters for object manager in main.cc.

* review feedback.

* use vector consturctor.

* linting

* remove profile visualizations.

* test fixes.

* linting.

* kill specific pids and use less memory.

* linting.

* simplify tests.

* Asynchronous IO for ObjectManager messages and object transfer.

* Revert "Asynchronous IO for ObjectManager messages and object transfer."

This reverts commit 4af43b159babc04daf80d1543e27c2cb46b7b19d.

* update test configuration to reflect changes in #1891

* review feedback.

* linting.
2018-04-14 17:08:19 -07:00
Melih ElibolandPhilipp Moritz 6a84b1f26e Remove num_threads as a parameter. (#1891)
* remove num_threads as a parameter.

* linting.

* add additional checks.

* Invoke TransferCompleted on failures.

* Fix issue with failed Gets on store.

* ray check status of writing object headers.

* fix mac issues.
2018-04-14 15:22:59 -07:00
Melih ElibolandPhilipp Moritz 6be73350c6 Adds Valgrind tests for multi-threaded object manager. (#1890)
* adds valgrind to new object manager.

* Add some comments.

* Update run_object_manager_valgrind.sh

typo

* Update run_object_manager_tests.sh

* update tests to reflect changes in #1891.

* reduce # tests.
2018-04-13 21:56:12 -07:00
Robert NishiharaandPhilipp Moritz 4379e9cea0 Pin cython version in docker base dependencies file. (#1898) 2018-04-13 20:33:20 -07:00
Robert NishiharaandPhilipp Moritz 24c944e499 Update arrow to efficiently serialize more types of numpy arrays. (#1889) 2018-04-13 09:41:51 -07:00
Eric LiangandGitHub e4b17e03f6 updates (#1896) 2018-04-13 00:57:00 -07:00
Peter SchafhalterandDevin Petersohn 1d605e8f8a [DataFrame] Inherit documentation from Pandas (#1727)
* Added _inherit_docstrings

* DataFrame documentation inherits from Pandas

* Fix formatting

* Replace hasattr and document properties

* Fix rebase

* Override documentation for groupby

* Override documentation for series

* Don't overwrite property docstrings

* Fix property __doc__ for python2
2018-04-12 20:30:19 -07:00
Robert NishiharaandPhilipp Moritz d0fffec2d0 Update arrow and parquet-cpp. (#1875)
* Update arrow.

* Fix bug.

* Cherry-pick commit for fixing parquet segfault.

* Update arrow and revert auto-releasing buffer commit.

* Remove parquet cherry-pick.
2018-04-12 16:17:12 -07:00
Alexey TumanovandPhilipp Moritz 39cf6ff6e1 raylet command line resource configuration plumbing (#1882)
* raylet command line resource configuration plumbing

* Small changes.
2018-04-12 02:37:15 -07:00
Alexey TumanovandPhilipp Moritz 85d3963172 use raylet for remote ray nodes (#1880) 2018-04-11 22:06:46 -07:00
Eric LiangandGitHub 4dc04374f6 [rllib] Propagate dim option to deepmind wrappers (#1876)
* updates

* updates
2018-04-11 21:38:06 -07:00
alvkao58andRichard Liaw 15a668dd12 [RLLib] DDPG (#1685) 2018-04-11 15:08:39 -07:00
Philipp MoritzandRobert Nishihara 74162d1492 Lint Python files with Yapf (#1872) 2018-04-11 10:11:35 -07:00
Omkar SalpekarandDevin Petersohn a3ddde398c [DataFrame] Fixed repr, info, and memory_usage (#1874)
* working with dataframes with too many rows and columns

* repr works for jupyter notebooks now

* added comments and test file

* added repr test file to .travis.yml

* added back ray.dataframe as pd to test file

* fixed pandas importing issues in test file

* getting the front and back of df more efficiently

* only keeping dataframe tests in travis

* fixing numpy array for row and col lengths issue

* doesn't add dimensions if df is small enough

* implemented memory_usage()

* completed memory_usage - still failing 2 tests

* only failing one test for memory_usage

* all repr and dataframes tests passing now

* fixing error related to python2 in info()

* fixing python2 errors

* fixed linting errosr

* using _arithmetic_helper in memory_usage()

* fixed last lint error

* removed testing-specific code

* adding back travis test

* removing extra tests from travis

* re-added concat test

* fixes with new indexing scheme

* code cleanup

* fully working with new indexing scheme

* added tests for info and memory_usage

* removed test file
2018-04-11 08:07:07 -07:00
Devin PetersohnandRobert Nishihara 806b2c844e Fix getattr compat (#1871) 2018-04-10 21:28:59 -07:00
alonamidandRobert Nishihara 202f9683ea check if arrow build dir exists (#1863) 2018-04-10 14:52:51 -07:00
Patrick YangandDevin Petersohn 521b549e4a [DataFrame] Encapsulate index and lengths into separate class (#1849)
* baseline impl for index_df.py

* added skeleton for index_df.py

* initial impl index_df

* separate out partition and non-partition impls

* add len function

* drop returns index_df slice of dropped indices

* housecleaning

* Integrate index overhaul

* Rename index df to index metadata

* Fix flake8 issues

* Addressing issues

* fix import issue

* Added metadata passing to constructor
2018-04-10 14:30:20 -07:00
Peter SchafhalterandDevin Petersohn 405b05d58a [DataFrame] Implemented __getattr__ (#1753)
* __getattr__ accesses columns

* Added test
2018-04-10 10:19:33 -07:00
Richard LiawandRobert Nishihara e82bea40b1 Add better analytics to docs (#1854) 2018-04-10 00:51:44 -07:00
adgirishandDevin Petersohn efeaacbedc Adding support for concat (#1739)
adding tests

fixing flake8

adding init

flake 8 on test

fixing tests, imports, and flake8

handling for index

adding tests for row, index

added more robust error handling for axis

fixing test failures

cleaning up error sfor 2.7

updating travis

resolving import

fixing flake8

moved import order

Fixing to refactor and delaying implementing ray-pd inner concat

resolving ray-pd concat and from_pandas mutation

Revert "resolving ray-pd concat and from_pandas mutation"

This reverts commit 5db43e4e89e328286532f3ef98a4526575c5d08d.
2018-04-09 21:36:24 -07:00
Philipp MoritzandRobert Nishihara 3039cca242 add facility to link libraries to tests (#1850) 2018-04-09 18:59:24 -07:00
Philipp MoritzandRobert Nishihara 834e594709 [XRay] Register object store and raylet with the GCS (#1860) 2018-04-09 18:56:33 -07:00
Robert NishiharaandPhilipp Moritz 7c9e291b4b In the UI, display task breakdowns by default. (#1857) 2018-04-09 13:24:38 -07:00
Jones WongandEric Liang 6b85d15bd9 supplement missed optimizer argument clip_rewards in default DQN configuration (#1853) 2018-04-08 17:58:05 -07:00
Robert NishiharaandStephanie Wang 256389dc59 Use new task spec for computing IDs in raylet code path. (#1830)
* Use new task spec for computing IDs in raylet code path.

* Fix linting.

* Fixes

* Fix test.
2018-04-08 13:31:55 -07:00
Robert NishiharaandPhilipp Moritz 0b7ad668ff Fix unused lambda capture compilation error. (#1844)
* Fix unused lambda capture compilation error.

* Fix linting.
2018-04-07 14:54:21 -07:00
Eric LiangandGitHub 79d37ce240 [rllib] Switch to use lz4 instead of snappy (#1847)
* lz4

* comment

* updates
2018-04-07 14:43:45 -07:00
Eric LiangandGitHub e6c00b2b5e [tune] Add util function to broadcast objects (#1845)
* add util

* Fri Apr  6 15:09:20 PDT 2018

* doc

* Fri Apr  6 15:21:42 PDT 2018

* Fri Apr  6 15:28:07 PDT 2018

* Fri Apr  6 15:28:26 PDT 2018

* Update tune-config.rst

* Update tune-config.rst
2018-04-07 11:37:14 -07:00
Richard LiawandGitHub bc8f62c947 [tune] Fix Median Stopping Rule Verbosity (#1833) 2018-04-06 22:58:13 -07:00
Stephanie WangandGitHub bef1d872b4 [xray] Cleanup Raylet processes on exit (#1839)
* Add raylet monitor script to timeout Raylet heartbeats

* Unit test for removing a different client from the client table

* Set node manager heartbeat according to global config

* Doc and fixes

* Add regression test for client table disconnect, refactor client table

* Convert 'Terminate' methods to destructors

* Destroy the Raylet on a SIGTERM

* Clean up workers on a SIGTERM
2018-04-06 17:21:51 -07:00
Melih ElibolandRobert Nishihara 3bf80839cb Remove all runtime errors. (#1840) 2018-04-06 17:20:52 -07:00
Melih ElibolandRobert Nishihara c7e11e9057 lint fix. (#1842) 2018-04-06 13:28:52 -07:00
Melih ElibolandGitHub 24a8cede88 Cache object info from store notification. (#1815)
Cache all object info from object added store notification & submit to GCS via object directory.
2018-04-06 02:33:23 -07:00
Stephanie WangandRobert Nishihara bf194db4bc [xray] Basic actor support (#1835) 2018-04-06 00:17:14 -07:00
Melih ElibolandGitHub 313b864e66 disconnect bug fix. (#1837) 2018-04-05 22:10:51 -07:00
Stephanie WangandRobert Nishihara cbf3181fd2 [xray] Monitor for Raylet processes (#1831)
* Add raylet monitor script to timeout Raylet heartbeats

* Unit test for removing a different client from the client table

* Set node manager heartbeat according to global config

* Doc and fixes

* Add regression test for client table disconnect, refactor client table

* Fix linting.
2018-04-05 20:45:38 -07:00
Devin PetersohnandRobert Nishihara 0d9a7a3c19 [DataFrame] Update architecture to be more flexible and performant (#1821) 2018-04-05 15:14:33 -07:00
Robert NishiharaandPhilipp Moritz 5bde5e75e7 Implement unsafe method for flushing entire object table and task table. (#1824)
* Implement unsafe method for flushing entire object table and task table.

* Add test.

* Fix test.
2018-04-04 18:29:24 -07:00
Richard LiawandGitHub 888e70f1be [tune] HyperOpt Support (v2) (#1763) 2018-04-04 11:08:26 -07:00
Alexey TumanovandPhilipp Moritz 5a9e83761d fix unused-lambda-capture on clang version 9.1 (#1823)
* fix unused-lambda-capture on clang9.1

* unused lambda capture fix continued

* lambda capture: NM

* lambda capture

* Fix linting.
2018-04-04 11:04:10 -07:00
Robert NishiharaandPhilipp Moritz e0193a5501 Print backtrace for RAY_LOG(FATAL) and also add file and line number … (#1805)
* Print backtrace for RAY_LOG(FATAL) and also add file and line number in common case.

* Fix linting.
2018-04-03 10:12:46 -07:00
Robert NishiharaandPhilipp Moritz fbfbb1c079 [xray] Integrate worker.py with raylet. (#1810)
* Integrate worker with raylet.

* Begin allowing worker to attach to cluster.

* Fix linting and documentation.

* Fix linting.

* Comment tests back in.

* Fix type of worker command.

* Remove xray python files and tests.

* Fix from rebase.

* Add test.

* Copy over raylet executable.

* Small cleanup.
2018-04-03 02:38:56 -07:00
Robert NishiharaandPhilipp Moritz 0fc989c6c1 Don't use 127.0.0.1 for local ip address. (#1596)
* Don't use 127.0.0.1 for ip address.

* Update test
2018-04-02 00:34:20 -07:00
Robert NishiharaandPhilipp Moritz d3e974a9a4 Increase ulimit -n in autoscaler examples. (#1769) 2018-04-02 00:32:56 -07:00
Robert NishiharaandPhilipp Moritz 27a0d58e54 Include resource string in error message for infeasible actors. (#1768) 2018-04-02 00:31:30 -07:00
Robert NishiharaandPhilipp Moritz 23b8793f0e Update documentation and autoscaler to find 0.4.0. (#1789) 2018-04-02 00:28:47 -07:00