Commit Graph

116 Commits

Author SHA1 Message Date
Eric Liang 0aa9373d62 Revert "Removing Pyarrow dependency (#7146)" (#7209)
This reverts commit 2116fd3bca.
2020-02-18 14:12:06 -08:00
ijrsvt 2116fd3bca Removing Pyarrow dependency (#7146) 2020-02-17 18:00:13 -08:00
Qing Wang 94a286ef1d [Java] Add session_dir as temp_dir for logs, socket files like Python (#7044)
* Support

* Add gcs_server support

* Fix ut

* Fix

* Remove unused py code

* Fix linting

* Fix cross language ci

* Fix CI

* Add docstring

* Fix

* Fix linting

* Add a singleton for config

* Refine

* fix

* Fix

* linting

* Remove FileUnit

* Fix

* Fix

* Fix

* Update java/runtime/src/main/java/org/ray/runtime/config/RayConfig.java

Co-Authored-By: Hao Chen <chenh1024@gmail.com>

* Fix streaming singleprocess CI

* Fix checkstyle

Co-authored-by: Hao Chen <chenh1024@gmail.com>
2020-02-13 17:49:52 +08:00
ijrsvt 0826f95e1c Including psutil & setproctitle (#7031) 2020-02-05 14:16:58 -08:00
fangfengbin ade7ebfc0c Add service based gcs client (#6686) 2020-02-05 12:06:25 +08:00
Richard Liaw 341ddd0a09 [tune] Default to TensorboardX and include in requirements. (#6836) 2020-01-19 01:49:33 -08:00
Mitchell Stern 763818b476 [Dashboard] Add static assets for speedscope v1.5.3 (#6822) 2020-01-17 20:53:53 -08:00
chaokunyang 4097d076d4 Package ray java jars into wheels (#6600) 2020-01-10 11:41:00 +08:00
Sven 60d4d5e1aa Remove future imports (#6724)
* Remove all __future__ imports from RLlib.

* Remove (object) again from tf_run_builder.py::TFRunBuilder.

* Fix 2xLINT warnings.

* Fix broken appo_policy import (must be appo_tf_policy)

* Remove future imports from all other ray files (not just RLlib).

* Remove future imports from all other ray files (not just RLlib).

* Remove future import blocks that contain `unicode_literals` as well.
Revert appo_tf_policy.py to appo_policy.py (belongs to another PR).

* Add two empty lines before Schedule class.

* Put back __future__ imports into determine_tests_to_run.py. Fails otherwise on a py2/print related error.
2020-01-09 00:15:48 -08:00
Chaokun Yang 7bbfa85c66 [Streaming] Streaming data transfer java (#6474) 2019-12-22 10:56:05 +08:00
Philipp Moritz 4d71ab83cf require packaging (#6517) 2019-12-17 12:01:14 -08:00
Philipp Moritz afae8406da Make sure numpy >= 1.16.0 is installed for fast pickling support (#6486)
* Make sure numpy >= 1.16.0 is installed

* Works for 1.15.4

* lint

* formatting

* update

* put check into the right place

* lint
2019-12-14 16:36:49 -08:00
alindkhare 76e678d775 [Serve] Added deadline awareness (#6442)
* [Serve] Added deadline awareness

Added deadline awareness while enqueuing a query
Using Blist sorted-list implementation (ascending order) to get queries according to their specified deadlines. [buffer_queues]
Exposed slo_ms via handle/http request
Added slo example 
The queries in example will be executed in almost the opposite order of which they are fired
Added slo pytest
Added check for slo_ms to not be negative
Included the changes suggested

* Linting Corrections

* Adding the code changes suggested by format.sh

* Added the suggested changes

Added justification for blist
Added blist in travis/ci/install-dependencies.sh

* Fixed linting issues

* Added blist to ray/doc/requirements-doc.txt
2019-12-11 16:41:54 -08:00
Chaokun Yang 6272907a57 [Streaming] Streaming data transfer and python integration (#6185) 2019-12-10 20:33:24 +08:00
Simon Mo 22b305223a Build Docker Containers for Linux Wheels (#6233) 2019-11-27 17:05:36 -08:00
Philipp Moritz decaa65cd6 Use pickle by default for serialization (#5978) 2019-11-10 18:12:18 -08:00
Philipp Moritz f7455839bf Expose raylet info to dashboard (#6045) 2019-10-31 17:36:59 -07:00
Ujval Misra a851d7eb87 [tune] Readable trial progress output (#5822)
* Cleaner, tabulated progress output.

* Minor HTML changes, trial ID instead of name

* Revert basic variant changes

* Cleanup, address richard's comments, add progress_reporter.py

* Add tabulate dependency

* Added more info to table, auto-hide columns with no data.

* lint

* Address comments

* Replace experiment tag w/ trial ID

* Fixed tests.

* Fixed test

* Added requirement

* Fix formatting
2019-10-08 16:38:39 -07:00
Simon Mo 9bb3633cd9 [Serve] Implement metric interface (#5852)
* Implement metric interface

* Address comment: made actor_handles a dict

* Fix iteration

* Lint

* Mark lightweight actors as num_cpus=0 to prevent resource starvation

* Be more explicit about the readiness condition

* Make task_runner non-blocking

* Lint
2019-10-07 09:29:26 -07:00
Simon Mo e8570874b6 [Serve] Implement flask_request and named python request (#5849)
* Implement flask_request and named python request

* Forgot to include missing files

* Address comment

* Add flask to requirements for doc (lint failed)

* Update doc requirement so lint will build

* Install flask in CI

* Fix typo in .travis.yml
2019-10-06 15:12:30 -07:00
Edward Oakes 972dddd776 [autoscaler] Kubernetes autoscaler backend (#5492)
* Add Kubernetes NodeProvider to autoscaler

* Split off SSHCommandRunner

* Add KubernetesCommandRunner

* Cleanup

* More config options

* Check if auth present

* More auth checks

* Better output

* Always bootstrap config

* All working

* Add k8s-rsync comment

* Clean up manual k8s examples

* Fix up submit.yaml

* Automatically configure permissisons

* Fix get_node_provider arg

* Fix permissions

* Fill in empty auth

* Remove ray-cluster from this PR

* No hard dep on kubernetes library

* Move permissions into autoscaler config

* lint

* Fix indentation

* namespace validation

* Use cluster name tag

* Remove kubernetes from setup.py

* Comment in example configs

* Same default autoscaling config as aws

* Add Kubernetes quickstart

* lint

* Revert changes to submit.yaml (other PR)

* Install kubernetes in travis

* address comments

* Improve autoscaling doc

* kubectl command in setup

* Force use_internal_ips

* comments

* backend env in docs

* Change namespace config

* comments

* comments

* Fix yaml test
2019-10-03 10:17:00 -07:00
Mitchell Stern b03147e7bf Update call to py-spy to conform to new API (#5758) 2019-09-23 14:52:23 -07:00
Mitchell Stern 98dcc1d440 [Dashboard] Add initial version of new dashboard (#5730) 2019-09-23 08:50:40 -07:00
Philipp Moritz a6dd794818 [Projects] Fix template path (#5716) 2019-09-16 19:58:54 -07:00
Simon Mo 5f88823c49 [Serve] Rewrite Ray.Serve From Scratch (#5562)
* Commit and format files

* address stylistic concerns

* Replcae "Usage" by "Example" in doc

* Rename srv to serve

* Add serve to CI process; Fix 3.5 compat

* Improve determine_tests_to_run.py

* Quick cosmetic for determien_tests

* Address comments

* Address comments

* Address comment

* Fix typos and grammar

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* Update python/ray/experimental/serve/global_state.py

Co-Authored-By: Edward Oakes <ed.nmi.oakes@gmail.com>

* Use __init__ for Query and WorkIntent class

* Remove dataclasses dependency

* Rename oid to object_id for clarity

* Rename produce->enqueue_request, consume->dequeue_request

* Address last round of comment
2019-09-13 21:36:56 -07:00
Devin Petersohn c33d6662ce Remove Modin from Ray wheels. (#5647)
There are several reasons for this:

* We no longer support python2
* There should be only 1 way of installing Modin
  * Issue management on these wheels
* I have never heard of anyone using this feature
* It is rarely kept up to date
* Modin depends on specific versions of Ray because of past API changes
2019-09-05 23:46:27 -07:00
Simon Mo d9b45cceec [Project] Implementing Project CLI (#5397) 2019-08-08 21:28:25 -07:00
Philipp Moritz e8d9cfc1f1 Ray projects schema and validation (#5329) 2019-08-06 14:36:04 -07:00
Simon Mo 196495a4de Fix Redis Test (#5302) 2019-07-30 00:22:16 -07:00
Hao Chen 8a30b93e42 Define common data structures with protobuf. (#5121) 2019-07-08 22:41:37 +08:00
Eric Liang 0448847a02 Update protobuf version (#5128) 2019-07-06 15:59:55 -07:00
Hao Chen 0131353d42 [gRPC] Migrate gcs data structures to protobuf (#5024) 2019-06-25 14:31:19 -07:00
Robert Nishihara d2f5b71c3b Remove typing from setup.py install_requirements. (#4971) 2019-06-12 15:02:12 -07:00
Philipp Moritz b0f6ddf6d1 Remove CMake files (#4493) 2019-04-02 22:17:33 -07:00
Philipp Moritz c5e2c9af4d Build wheels for macOS with Bazel (#4280) 2019-03-15 10:37:57 -07:00
Richard Liaw 6630a35353 [tune] Initial Commit for Tune CLI (#3983)
This introduces a light CLI for Tune.
2019-03-08 16:46:05 -08:00
Philipp Moritz 95254b3d71 Remove the old web UI (#4301) 2019-03-07 23:15:11 -08:00
Hao Chen f0465bc68c [Java] Refine tests and fix single-process mode (#4265) 2019-03-07 09:59:13 +08:00
Robert Nishihara 387c98cf01 Make sure dashboard is packaged with wheels. (#4175) 2019-02-27 18:36:49 -08:00
Yuhong Guo 41b81af11b Downgrade six to 1.0.0 (#4180) 2019-02-27 13:05:25 -08:00
Philipp Moritz ba52caff37 Make Bazel the default build system (#3898) 2019-02-23 11:58:59 -08:00
Daniel Edgecumbe 2e30f7ba38 Add a web dashboard for monitoring node resource usage (#4066) 2019-02-21 00:10:04 -08:00
Yuhong Guo 1f864a02bc Add option of load_code_from_local which is required in cross-language ray call. (#3675) 2019-02-21 12:37:17 +08:00
Philipp Moritz cfc7e2c5a9 Fix modin test (#4069) 2019-02-18 12:17:36 -08:00
Si-Yuan 48139cf861 Migrate Python C extension to Cython (#3541) 2019-01-24 09:17:14 -08:00
Robert Nishihara 19908c01b8 Use environment markers to only install faulthandler in Python < 3.3. (#3764) 2019-01-14 15:55:59 +08:00
nam-cern 3d8f56409b Ensure numpy is at least 1.10.4 in setup.py (#2462)
In the build script, numpy is specifically set at 1.10.4. We should also ensure that it is indeed the case in `setup.py`.
2018-12-24 11:01:25 -08:00
Robert Nishihara a5309bec7c Make README render properly on PyPI. (#3578)
* Make README render properly in pypi.

* Add small logo

* temporary fix

* smaller image

* Remove image size.

* Add author and email to setup.py.
2018-12-19 18:41:09 -08:00
Philipp Moritz b3bf608608 Update arrow to reduce plasma IPCs. (#3497) 2018-12-14 23:49:37 -05:00
Si-Yuan 84fae57ab5 Convert the raylet client (the code in local_scheduler_client.cc) to proper C++. (#3511)
* refactoring

* fix bugs

* create client class

* create client class for java; bug fix

* remove legacy code

* improve code by using std::string, std::unique_ptr rename private fields and removing legacy code

* rename class

* improve naming

* fix

* rename files

* fix names

* change name

* change return types

* make a mutex private field

* fix comments

* fix bugs

* lint

* bug fix

* bug fix

* move too short functions into the header file

* Loose crash conditions for some APIs.

* Apply suggestions from code review

Co-Authored-By: suquark <suquark@gmail.com>

* format

* update

* rename python APIs

* fix java

* more fixes

* change types of cpython interface

* more fixes

* improve error processing

* improve error processing for java wrapper

* lint

* fix java

* make fields const

* use pointers for [out] parameters

* fix java & error msg

* fix resource leak, etc.
2018-12-13 13:39:10 -08:00