Commit Graph

84 Commits

Author SHA1 Message Date
Siyuan (Ryans) Zhuang b68fede30b Convert include guard to pragma once (#8957) 2020-06-16 01:29:43 -07:00
chaokunyang cb6f337372 [Java] Refine python function (#8943) 2020-06-16 16:22:49 +08:00
mehrdadn 101c215125 Get more tests running on Windows (#6537)
* Get rid of system() calls

* Work around '/usr/share/mini' showing up on GitHub Actions (probably due to psutil truncation)

https://github.com/ray-project/ray/runs/722480047?check_suite_focus=true

* Don't check for socket max path length on Windows

* Don't check for socket existence on Windows

* Fix race condition in Windows fate-sharing

* Work around missing .exe extension for Redis tests

* Add more tests to GitHub Actions

Co-authored-by: Mehrdad <noreply@github.com>
2020-06-12 21:32:10 -07:00
chaokunyang dfa4768fc6 [Java] Refactor java api (#8858) 2020-06-12 10:49:01 +08:00
chaokunyang 700d81fa20 [Java] Remove java api sub package from test module (#8853) 2020-06-11 14:59:45 +08:00
Tianyi Chen ec5ecb661f [Streaming] Implement streaming job-worker. (#8780) 2020-06-10 14:13:55 +08:00
chaokunyang d2ef29f0d2 [Streaming] Fix streaming word count tests (#8697) 2020-06-09 11:49:19 +08:00
chaokunyang 31a4d07bc4 [Java] Rename java ObjectRef/ActorHandle (#8799) 2020-06-09 11:40:43 +08:00
chaokunyang d04953ab3c [Streaming] Union api (#8612) 2020-06-08 14:28:11 +08:00
Siyuan (Ryans) Zhuang ea05ebe89e Ship plasma store with Ray (#7901) 2020-06-03 17:44:34 -07:00
Alec Brickner 207ab44129 Raise major version limit for msgpack (#8466) 2020-06-01 20:00:36 -07:00
chaokunyang bcdbe2d3d4 Streaming rich function (#8602) 2020-05-27 18:36:07 +08:00
Tao Wang a1298686d7 [TEST]Use manager class to start/stop components instead of spreading duplicated codes everywhere (#8500) 2020-05-27 16:51:51 +08:00
chaokunyang 8da00af74e [Streaming ]print streaming error when tests fail (#8594) 2020-05-27 11:04:08 +08:00
Kai Yang 2e5e789294 Allow enabling logging in core worker with empty log_dir (#8529) 2020-05-22 18:02:37 +08:00
Stephanie Wang bd169749e0 Option to retry failed actor tasks (#8330)
* Python

* Consolidate state in the direct actor transport, set the caller starts at

* todo

* Remove unused

* Update and unit tests

* Doc

* Remove unused

* doc

* Remove debug

* Update src/ray/core_worker/transport/direct_actor_transport.h

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* Update src/ray/core_worker/transport/direct_actor_transport.cc

Co-authored-by: Eric Liang <ekhliang@gmail.com>

* lint and fix build

* Update

* Fix build

* Fix tests

* Unit test for max_task_retries=0

* Fix java?

* Fix bad test

* Cross language fix

* fix java

Co-authored-by: Eric Liang <ekhliang@gmail.com>
2020-05-15 20:15:15 -07:00
Max Fitton 00325eb2b2 Rename max_reconstructions to max_restarts and use -1 for infinite (#8274)
Co-authored-by: Edward Oakes <ed.nmi.oakes@gmail.com>
2020-05-14 10:30:29 -05:00
Edward Oakes 2677b71003 Implement named actors using the GCS service (#8328) 2020-05-09 08:58:10 -05:00
mehrdadn 4bdef78e2e Various CI fixes and cleanup (#8289) 2020-05-05 10:47:49 -07:00
mehrdadn 254b1ec370 Set up testing and wheels for Windows on GitHub Actions (#8131)
* Move some Java tests into ci.sh

* Move C++ worker tests into ci.sh

* Define run()

* Prepare to move Python tests into ci.sh

* Fix issues in install-dependencies.sh

* Reload environment for GitHub Actions

* Move wheels to ci.sh and fix related issues

* Don't bypass failures in install-ray.sh anymore

* Make CI a little quieter

* Move linting into ci.sh

* Add vitals test right after build

* Fix os.uname() unavailability on Windows

Co-authored-by: Mehrdad <noreply@github.com>
2020-04-29 21:19:02 -07:00
chaokunyang 91f630f709 [Streaming] Streaming Cross-Lang API (#7464) 2020-04-29 13:42:08 +08:00
WuTao 32c2055c99 Streaming state (#7348) 2020-04-28 10:36:32 +08:00
chaokunyang 5cf49d5edd Fix streaming ci (#8159) 2020-04-26 20:56:58 +08:00
ijrsvt 69ff7e3e35 TaskCancellation (#7669)
* Smol comment

* WIP, not passing ray.init

* Fixed small problem

* wip

* Pseudo interrupt things

* Basic prototype operational

* correct proc title

* Mostly done

* Cleanup

* cleaner raylet error

* Cleaning up a few loose ends

* Fixing Race Conds

* Prelim testing

* Fixing comments and adding second_check for kill

* Working_new_impl

* demo_ready

* Fixing my english

* Fixing a few problems

* Small problems

* Cleaning up

* Response to changes

* Fixing error passing

* Merged to master

* fixing lock

* Cleaning up print statements

* Format

* Fixing Unit test build failure

* mock_worker fix

* java_fix

* Canel

* Switching to Cancel

* Responding to Review

* FixFormatting

* Lease cancellation

* FInal comments?

* Moving exist check to CoreWorker

* Fix Actor Transport Test

* Fixing task manager test

* chaning clock repr

* Fix build

* fix white space

* lint fix

* Updating to medium size

* Fixing Java test compilation issue

* lengthen bad timeouts
2020-04-25 16:04:52 -07:00
Tianyi Chen 0204dff1e9 [streaming]Add master and scheduler. (#8044) 2020-04-22 14:43:56 +08:00
mehrdadn c8b9a357f2 Try to fix dependency issue (#8065)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-19 16:09:29 -07:00
mehrdadn 8cf37726d2 Fix missing Java dependency (#8067)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 10:43:02 -05:00
mehrdadn f15618033d Remove --no-transfer-progress as it appears to be unsupported (#8066)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 16:30:37 +02:00
Richard Liaw 118d960e1c [hotfix] Java Lint Broken (#8048) 2020-04-16 13:58:33 -07:00
Clark Zinzow d4cae5f632 [Core] Added ability to specify different IP addresses for a core worker and its raylet. (#7985) 2020-04-16 10:32:24 -05:00
wanxing 9345d03ffb [Streaming] Streaming data transfer supports cross language. (#7961)
* add init parameters for java

* fix bug

* cython

* fix compile

* fix test_direct_tranfer

* comment

* ChannelCreationParameter

* fix comment

* builder

* lint and fix tests

* fix single process test

* fix checkstyle and lint

* checkstyle

* lint python

Co-authored-by: wanxing <wanxing@B-458DMD6M-1753.local>
2020-04-16 15:16:48 +08:00
JianZhangYang 7b0518b993 [streaming] Async changes for resourcemanager part (#7955) 2020-04-16 14:15:45 +08:00
mehrdadn ba00c29b67 Factor out Travis 'install' sections for use with GitHub Actions (#7988) 2020-04-15 08:10:22 -07:00
Qing Wang 98bfcd53bc [Java] Rename group id and package name. (#7864)
* Initial

* Change streaming's

* Fix

* Fix

* Fix org_ray

* Fix cpp file name

* Fix streaming

* Fix

* Fix

* Fix testlistening

* Fix missing sth in python

* Fix

* Fix

* Fix SPI

* Fix

* Fix complation

* Fix

* Fix CI

* Fix checkstyle

Fix checkstyle

* Fix streaming tests

* Fix streaming CI

* Fix streaming checkstyle.

* Fix build

* Fix bazel dep

* Fix

* Fix ray checkstyle

* Fix streaming checkstyle

* Fix bazel checkstyle
2020-04-12 17:59:34 +08:00
mehrdadn 4aa68b82fa [CI] Various Improvements to Travis Scripts (#7956)
* Delete LINT section of install-ray.sh since it appears unused

* Delete install.sh since it appears unused

* Delete run_test.sh since it appears unused

* Put environment variables on separate lines in .travis.yml

* Move --jobs 50 out of install-ray.sh

* Delete upgrade-syn.sh since it appears unused

* Move CI bazel flags to .bazelrc via --config

* Make installations quieter

* Get rid of verbose Maven messages

* Install Bazel system-wide for CI so that there's no need to update PATH

* Recognize Windows as valid platform

Co-authored-by: Mehrdad <noreply@github.com>
2020-04-10 13:26:28 -07:00
Lingxuan Zuo 0d713e3eba [Streaming] Try to trigger mock transfer tests ci (#7885)
* try to trigger mock transfer tests ci

* execute transfer tests

* show all logs when bazel test streaming

* temporary repeated ci runs

* Revert "temporary repeated ci runs"

This reverts commit dc77d2f9f79b5fa7b490221a8e9089e6349e067d.
2020-04-10 11:56:59 +08:00
fangfengbin 061043229f [GCS]Optimize gcs client testcases (#7895) 2020-04-09 12:30:58 +08:00
Tianyi Chen c5bf9cc472 [streaming] Sync changes for graph part. (#7827) 2020-04-09 12:30:44 +08:00
fyrestone fc6259a656 Cross language serialization for primitive types (#7711)
* Cross language serialization for Java and Python

* Use strict types when Python serializing

* Handle recursive objects in Python; Pin msgpack >= 0.6.0, < 1.0.0

* Disable gc for optimizing msgpack loads

* Fix merge bug

* Java call Python use returnType; Fix ClassLoaderTest

* Fix RayMethodsTest

* Fix checkstyle

* Fix lint

* prepare_args raises exception if try to transfer a non-deserializable object to another language

* Fix CrossLanguageInvocationTest.java, Python msgpack treat float as double

* Minor fixes

* Fix compile error on linux

* Fix lint in java/BUILD.bazel

* Fix test_failure

* Fix lint

* Class<?> to Class<T>; Refine metadata bytes.

* Rename FST to Fst; sort java dependencies

* Change Class<?>[] to Optional<Class<?>>; sort requirements in setup.py

* Improve CrossLanguageInvocationTest

* Refactor MessagePackSerializer.java

* Refactor MessagePackSerializer.java; Refine CrossLanguageInvocationTest.java

* Remove unnecessary dependencies for Java; Add getReturnType() for RayFunction in Java

* Fix bug

* Remove custom cross language type support

* Replace Serializer.Meta with MutableBoolean

* Remove @SuppressWarnings support from checkstyle.xml; Add null test in CrossLanguageInvocationTest.java

* Refine MessagePackSerializer.pack

* Ray.get support RayObject as input

* Improve comments and error info

* Remove classLoader argument from serializer

* Separate msgpack from pickle5 in Python

* Pair<byte[], MutableBoolean> to Pair<byte[], Boolean>

* Remove public static <T> T get(RayObject<T> object), use RayObject.get() instead

* Refine test

* small fixes

Co-authored-by: 刘宝 <po.lb@antfin.com>
Co-authored-by: Hao Chen <chenh1024@gmail.com>
2020-04-08 21:10:57 +08:00
Kai Yang 48b48cc8c2 Support multiple core workers in one process (#7623) 2020-04-07 11:01:47 +08:00
Qing Wang 1b477c6dd0 [Streaming] Fix streaming CI failure. (#7869) 2020-04-03 10:45:25 +08:00
Tianyi Chen f889f938e5 [streaming] Use enum to define resource type. (#7813) 2020-03-31 00:03:49 +08:00
Robert Nishihara ee8c9ff732 Remove six and cloudpickle from setup.py. (#7694) 2020-03-23 11:42:05 -07:00
Edward Oakes 8b4f5a9431 Remove non-direct-call code from core worker (#7625) 2020-03-22 19:20:08 -05:00
Edward Oakes 58dc70f90e [minor] Remove get_global_worker(), RuntimeContext (#7638) 2020-03-20 15:45:29 -05:00
fyrestone a1ae935839 Java call Python use structured function descriptors (#7634) 2020-03-20 17:29:45 +08:00
Kai Yang 6b888b0247 [Java] Make both RayActor and RayPyActor inheriting from BaseActor (#7462) 2020-03-17 21:45:56 +08:00
fyrestone 7697ea2be2 Java call Python actor method use actor.call (#7614) 2020-03-17 14:52:43 +08:00
mehrdadn a0700e2f86 Change /tmp to platform-specific temporary directory (#7529) 2020-03-16 18:10:14 -07:00
mehrdadn a87199d240 Fix cyclic dependency between ray/util and ray/common (#7581)
* Fix cyclic dependency

Headers in ray/util should not depend on those in ray/common

* Move random generations to ray/common/test_util.h

* Add license header

Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Philipp Moritz <pcmoritz@gmail.com>
2020-03-14 12:44:53 -07:00