Commit Graph
100 Commits
Author SHA1 Message Date
mehrdadnandMehrdad 5331c30e35 Improve Clang-IWYU to automatically make #include fixes (#9858)
Co-authored-by: Mehrdad <noreply@github.com>
2020-08-10 12:49:58 -07:00
mehrdadnandMehrdad b62ec7787f Ignore grep exit code for shellcheck in format.sh (#9861)
Co-authored-by: Mehrdad <noreply@github.com>
2020-08-02 00:59:05 -07:00
mehrdadnandMehrdad 78995d085f Fix macOS incompatibility in format.sh (#9832)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-31 09:25:55 -07:00
mehrdadnandMehrdad e2c0174ab2 Factor out some Bazel options into .bazelrc (#9804)
* Factor out --keep_going in Bazel --config=ci

* Remove Bazel --test_timeout=600 for Windows

* Use global --test_output for Bazel CI

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-30 18:09:31 -07:00
mehrdadn a7b97b6f8a Add shellcheck support (#8574) 2020-07-30 18:39:28 -05:00
mehrdadnandMehrdad 07022f3f11 Fix src/ray/core_worker/common.h deleted constructor (#9785)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-29 15:49:02 -07:00
mehrdadnandMehrdad fb5280f21b Fix some Windows CI issues (#9708)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-28 18:10:23 -07:00
mehrdadnandMehrdad 2949c09ee8 Fix remote-watch.py (#9625)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-27 15:54:23 -07:00
mehrdadnandMehrdad 56d2cf6479 Shellcheck rewrites (#9597)
* Fix SC2001: See if you can use ${variable//search/replace} instead.

* Fix SC2010: Don't use ls | grep. Use a glob or a for loop with a condition to allow non-alphanumeric filenames.

* Fix SC2012: Use find instead of ls to better handle non-alphanumeric filenames.

* Fix SC2015: Note that A && B || C is not if-then-else. C may run when A is true.

* Fix SC2028: echo may not expand escape sequences. Use printf.

* Fix SC2034: variable appears unused. Verify use (or export if used externally).

* Fix SC2035: Use ./*glob* or -- *glob* so names with dashes won't become options.

* Fix SC2071: > is for string comparisons. Use -gt instead.

* Fix SC2154: variable is referenced but not assigned

* Fix SC2164: Use 'cd ... || exit' or 'cd ... || return' in case cd fails.

* Fix SC2188: This redirection doesn't have a command. Move to its command (or use 'true' as no-op).

* Fix SC2236: Use -n instead of ! -z.

* Fix SC2242: Can only exit with status 0-255. Other data should be written to stdout/stderr.

* Fix SC2086: Double quote to prevent globbing and word splitting.

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 17:24:19 -05:00
mehrdadnandMehrdad 7344fb9858 Switch from GitHub checkout@v2 to checkout@v1 due to bugs in checkout (#9697)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 14:25:58 -07:00
mehrdadnandMehrdad 6beed5a2df Fix ERROR logging not being printed to standard error (#9633)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-24 11:15:00 -07:00
mehrdadnandMehrdad 75b2fc4313 Auto-cancel build when a new commit is pushed (#8043)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-23 16:07:00 -07:00
mehrdadnandMehrdad 6346c70792 Fix bug in Bazel version check (#9626)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 20:58:46 -07:00
mehrdadnandMehrdad b14728d999 Shellcheck quoting (#9596)
* Fix SC2006: Use $(...) notation instead of legacy backticked `...`.

* Fix SC2016: Expressions don't expand in single quotes, use double quotes for that.

* Fix SC2046: Quote this to prevent word splitting.

* Fix SC2053: Quote the right-hand side of == in [[ ]] to prevent glob matching.

* Fix SC2068: Double quote array expansions to avoid re-splitting elements.

* Fix SC2086: Double quote to prevent globbing and word splitting.

* Fix SC2102: Ranges can only match single chars (mentioned due to duplicates).

* Fix SC2140: Word is of the form "A"B"C" (B indicated). Did you mean "ABC" or "A\"B\"C"?

* Fix SC2145: Argument mixes string and array. Use * or separate argument.

* Fix SC2209: warning: Use var=$(command) to assign output (or quote to assign string).

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 21:56:41 -05:00
mehrdadn 4f470c3fc1 Shellcheck comments (#9595) 2020-07-21 16:47:09 -05:00
mehrdadnandMehrdad a2f31195c6 Fix Lint in setup.py (#9618)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 14:24:24 -07:00
mehrdadnandMehrdad c5cde65bc6 Add bazel to the PATH in setup.py (#9590)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-21 13:35:29 -07:00
mehrdadn 980ade01c8 Rename path variable due to zsh conflict (#9610) 2020-07-21 10:17:50 -07:00
mehrdadn f3ef9060e4 Handle warnings in core (#9575) 2020-07-20 12:55:07 -07:00
mehrdadnandMehrdad 3d0a3c47a8 Fix Windows CI (#9588)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-20 12:47:27 -07:00
mehrdadnandMehrdad 02cbd4fb7e Skip uneeded steps on CI (#9582)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-20 12:41:57 -07:00
mehrdadnandMehrdad dcec26ac7b Fix log losses (#9559)
* Close log on shutdown

* Disable log buffering

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-20 11:03:58 -07:00
mehrdadnandMehrdad bc842a7888 Fix setup.py bug (#9581)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-20 15:04:35 +02:00
mehrdadn 2554a1a997 Bazel fixes (#9519) 2020-07-19 12:53:08 -07:00
mehrdadnandMehrdad ffb88cda6f Fix Bazel in Docker (#9530)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-17 11:39:38 -07:00
mehrdadnandMehrdad 37942ea1e7 Windows cleanup (#9508)
* Remove unneeded code for Windows

* Get rid of usleep()

* Make platform_shims includes non-transitive

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-17 02:08:15 -07:00
mehrdadnandMehrdad aa33aba1fa Fix pip and Bazel interaction messing up CI (#9506)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-16 16:28:37 -07:00
mehrdadn ac39e23145 Get rid of build shell scripts and move them to Python (#6082) 2020-07-16 11:26:47 -05:00
mehrdadnandMehrdad 06ed2313e2 Fix clang-cl build (#9494)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-15 22:17:11 -07:00
mehrdadn aa8928fac2 Make more tests compatible with Windows (#9303) 2020-07-15 11:34:33 -05:00
mehrdadnandMehrdad ad83337f46 Make pip install verbose (#9496)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-15 18:02:42 +02:00
mehrdadnandMehrdad 33e400998c Fix name clash on Windows (#9412)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-14 23:14:53 -07:00
mehrdadnandMehrdad ca4f3b79db Speedups for GitHub Actions (#9343)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-14 14:51:51 -07:00
mehrdadnandMehrdad 3d65682e62 Bazel selects compiler flags based on compiler (#9313)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-13 15:31:46 -07:00
mehrdadnandMehrdad 5291bf235b TRAVIS_PULL_REQUEST is false for non-PRs, not empty (#9439)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-13 14:52:40 +02:00
mehrdadn 5c853eaa6a Fix copy to workspace (#9400) 2020-07-11 14:27:56 +02:00
mehrdadnandMehrdad dd2cc6eb48 Update hiredis and remove Windows patches (#9289)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-09 18:45:44 -07:00
mehrdadnandMehrdad 4687b807c4 Combine different severities into the same log files (#9230)
* Combine different severities into the same log files

Co-authored-by: Mehrdad <noreply@github.com>
2020-07-09 14:14:28 -07:00
mehrdadnandMehrdad b6c11f3dd7 Bazel move more shell to native rules (#9314)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-09 09:02:35 -07:00
mehrdadnandMehrdad 05d4cced4e [CI] GHA path with spaces (#9356)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-08 10:44:37 -07:00
mehrdadnandMehrdad 28d18ec836 [windows] Improve GPU detection (#9300)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-04 13:20:23 -07:00
mehrdadnandMehrdad 65f89753b0 Move full build to install_ray() (#9276)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-02 21:06:31 -07:00
mehrdadnandMehrdad 7135cb2aec Fix .exe file extensions (#9197)
Co-authored-by: Mehrdad <noreply@github.com>
2020-07-02 15:29:34 -05:00
29acf272b7 Build with Visual C++ (#9190)
Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Simon Mo <xmo@berkeley.edu>
2020-07-02 09:34:24 -07:00
mehrdadn 882f60012f Fix WSL patch (it doesn't fall under "win32") (#9195) 2020-06-29 21:00:00 -05:00
mehrdadnandMehrdad 79c4c67ed9 Split hiredis from Redis and use it directly (#9176)
* Split hiredis from Redis and use it directly

Co-authored-by: Mehrdad <noreply@github.com>
2020-06-29 18:16:32 -07:00
mehrdadnandMehrdad e060d99c6e Bazel: strip_prefix=True should strip leading 'v' (#9175)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-29 11:27:56 -07:00
mehrdadn bc140b69cc Patch WSL psutil bug (#6808) 2020-06-29 10:55:43 -05:00
mehrdadnandMehrdad f4e5561ef7 Fix build error (#9185)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-29 14:51:01 +08:00
mehrdadn 8f72a9fd1a Fix more Windows tests (#9173) 2020-06-28 17:13:15 +08:00
mehrdadnandMehrdad aac40af948 Fix some nonstandard code (#9172)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-28 00:22:52 -07:00
mehrdadnandMehrdad 898e472425 Make test_utils.py use pipes to avoid file access conflicts on Windows (#9072)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-27 22:33:45 +02:00
mehrdadnandMehrdad 11bda7272b [windows] Work around msgpack issue on PowerPC64LE (#9140)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-25 20:36:22 -07:00
mehrdadn ab3413c124 Make process termination more robust (#9081) 2020-06-25 15:30:53 -05:00
mehrdadnandMehrdad 07655036d2 [tune] os.replace() instead of os.rename() for cross-platform (#9141)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-25 12:30:51 -07:00
0487c250e8 Windows compatibility (#93)
Co-authored-by: mehrdadn <mehrdadn@users.noreply.github.com>
Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
2020-06-24 11:27:28 -07:00
mehrdadnandMehrdad 98d68b31c3 Suppress GRPC SO_REUSADDR warning on Linux (#7972)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-23 21:58:43 -07:00
mehrdadn 275da2e400 Fix Google log directory again (#9063) 2020-06-22 14:56:28 -05:00
mehrdadnandMehrdad 1a40d24174 Handle loop_ NULL case (#9067)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-22 11:05:29 -07:00
mehrdadnandMehrdad fc4684d3ca Update pandas to 1.0.5 (#9065)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-21 14:35:25 -05:00
mehrdadnandMehrdad 981f67bfb0 Fix more Windows issues (#9011)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-19 18:51:45 -07:00
mehrdadnandMehrdad f8d49d69c1 Fix and merge asio client read/write operations (#9026)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-19 18:49:55 -07:00
mehrdadn f43cad6371 Fix Google log directory (#9025) 2020-06-19 11:00:02 -05:00
92f67cd2ae Add Optional Fast Build Configuration (#8925)
* Fast builds by default

* Update doc/source/development.rst

Co-authored-by: Simon Mo <xmo@berkeley.edu>

Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Simon Mo <xmo@berkeley.edu>
2020-06-18 14:12:12 -07:00
mehrdadnandMehrdad 003cec87b4 [CI] Fix Conda Permission on MacOS Github Action(#9004)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-18 00:00:20 -07:00
mehrdadn 4afa2b304a Clean up CI ASAN & .bazelrc (#8828) 2020-06-15 17:27:17 -07:00
mehrdadnandMehrdad 101c215125 Get more tests running on Windows (#6537)
* Get rid of system() calls

* Work around '/usr/share/mini' showing up on GitHub Actions (probably due to psutil truncation)

https://github.com/ray-project/ray/runs/722480047?check_suite_focus=true

* Don't check for socket max path length on Windows

* Don't check for socket existence on Windows

* Fix race condition in Windows fate-sharing

* Work around missing .exe extension for Redis tests

* Add more tests to GitHub Actions

Co-authored-by: Mehrdad <noreply@github.com>
2020-06-12 21:32:10 -07:00
mehrdadn 07637e5b5b Upgrade Bazel and add required patches (#8847) 2020-06-12 14:59:22 -05:00
mehrdadnandMehrdad cae475c46a Fix Windows build (#8905)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-11 14:54:37 -07:00
mehrdadnandMehrdad 4bda333412 Fix Windows build issue introduced in #8852 (#8869)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-09 19:21:41 -07:00
50f6272fcc Replace ps call with psutil (#8851)
* Replace ps call with psutil

* Minor formatting

Co-authored-by: Mehrdad <noreply@github.com>
Co-authored-by: Robert Nishihara <robertnishihara@gmail.com>
2020-06-09 14:21:19 -07:00
mehrdadnandMehrdad 67cd9a0c95 CI fixes and improvements (#8849)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-09 14:07:54 -07:00
mehrdadnandMehrdad f007bfb4cf Merge Jenkins workflow into main workflow (#8848)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-09 11:07:14 -07:00
mehrdadnandMehrdad b8cc9a1cbb Change GitHub Actions path to one with spaces (#8850)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-08 23:48:02 -07:00
mehrdadnandMehrdad f93bb008bb Change os.uname()[1] and socket.gethostname() to the portable and faster platform.node_ip() (#8839)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-08 21:29:46 -07:00
mehrdadnandMehrdad 3ee2e9f7e5 Make #include consistent (#8666)
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-07 15:43:24 +02:00
mehrdadn f68183d778 Error-checking for a couple of corruption issues (#8059)
* Extra error handling
* Handle connection closed in Redis monitor
Co-authored-by: Mehrdad <noreply@github.com>
2020-06-07 15:43:00 +02:00
mehrdadn d78757623d bazel build --compilation_mode=debug (#6457) 2020-06-05 14:36:10 +02:00
mehrdadn cb91fe2fc4 SetErrorMode for all Ray processes (#8656) 2020-05-29 10:18:20 -05:00
mehrdadn 79a4eac48c Make more tests run on Windows (#8553) 2020-05-26 18:43:34 -05:00
mehrdadnandMehrdad ebf060d484 Make more tests run on Windows (#8446)
* Remove worker Wait() call due to SIGCHLD being ignored

* Port _pid_alive to Windows

* Show PID as well as TID in glog

* Update TensorFlow version for Python 3.8 on Windows

* Handle missing Pillow on Windows

* Work around dm-tree PermissionError on Windows

* Fix some lint errors on Windows with Python 3.8

* Simplify torch requirements

* Quiet git clean

* Handle finalizer issues

* Exit with the signal number

* Get rid of wget

* Fix some Windows compatibility issues with tests

Co-authored-by: Mehrdad <noreply@github.com>
2020-05-20 12:25:04 -07:00
mehrdadn 8da084bc54 Try to address linting issues (#8485) 2020-05-19 10:29:17 -05:00
mehrdadnandMehrdad c8cd716295 Restore unset -f cd (#8467)
Co-authored-by: Mehrdad <noreply@github.com>
2020-05-16 09:54:59 -07:00
mehrdadnandMehrdad cd0037064c Windows wheels for multiple Python versions (#8369)
* Upload wheels to latest directory as well on GitHub Actions

* Fix bug in install-dependencies.sh

* Move out bazel build //:* from install_ray, since it isn't really necessary for that purpose

* Build wheels for different versions of Python on Windows

* Compile Windows in opt mode

Co-authored-by: Mehrdad <noreply@github.com>
2020-05-12 22:06:04 -07:00
mehrdadn ac1ed293e3 Patch redis-py bug for Windows (#8386) 2020-05-12 10:41:45 -05:00
mehrdadnandMehrdad a3b95d4664 Make Travis clone the full repo and the exact commit requested (#8331)
Co-authored-by: Mehrdad <noreply@github.com>
2020-05-12 10:40:45 -05:00
mehrdadnandMehrdad 66b3edccb9 Prefer built-in system compilers over Clang download (#8355)
Co-authored-by: Mehrdad <noreply@github.com>
2020-05-11 11:53:35 -05:00
mehrdadn 4bdef78e2e Various CI fixes and cleanup (#8289) 2020-05-05 10:47:49 -07:00
mehrdadnandMehrdad ff68fb8c7c Try to fix tests running all the time (#8280)
Co-authored-by: Mehrdad <noreply@github.com>
2020-05-02 15:37:52 -05:00
mehrdadnandMehrdad bf074073e7 Deploy Windows wheels to Amazon S3 (#8237)
* Deploy to Amazon S3

* Install specifically requested Python version

Co-authored-by: Mehrdad <noreply@github.com>
2020-05-01 14:08:57 -07:00
mehrdadnandMehrdad 254b1ec370 Set up testing and wheels for Windows on GitHub Actions (#8131)
* Move some Java tests into ci.sh

* Move C++ worker tests into ci.sh

* Define run()

* Prepare to move Python tests into ci.sh

* Fix issues in install-dependencies.sh

* Reload environment for GitHub Actions

* Move wheels to ci.sh and fix related issues

* Don't bypass failures in install-ray.sh anymore

* Make CI a little quieter

* Move linting into ci.sh

* Add vitals test right after build

* Fix os.uname() unavailability on Windows

Co-authored-by: Mehrdad <noreply@github.com>
2020-04-29 21:19:02 -07:00
mehrdadnandMehrdad b9de9dadd7 Fix Windows build (#8186)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-26 13:07:25 -07:00
mehrdadn 0a54407961 [CI] Factor out more Travis code and update GitHub Actions (#8085) 2020-04-21 09:53:08 -07:00
mehrdadnandMehrdad c8b9a357f2 Try to fix dependency issue (#8065)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-19 16:09:29 -07:00
mehrdadnandMehrdad 8cf37726d2 Fix missing Java dependency (#8067)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 10:43:02 -05:00
mehrdadnandMehrdad f15618033d Remove --no-transfer-progress as it appears to be unsupported (#8066)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-17 16:30:37 +02:00
mehrdadnandMehrdad 35ae7f0e68 [CI] Preload Test to Skip Env Var to All Travis Job (#8061)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-16 15:37:25 -07:00
mehrdadnandMehrdad 42f88ecf9d Hotfix CI Export Tests to Skip (#8058)
Co-authored-by: Mehrdad <noreply@github.com>
2020-04-16 15:23:00 -07:00
mehrdadn 956ea7c944 Hotfix CI determine_tests_to_run (#8039) 2020-04-15 17:00:38 -07:00
mehrdadn ba00c29b67 Factor out Travis 'install' sections for use with GitHub Actions (#7988) 2020-04-15 08:10:22 -07:00