wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-01 10:52:54 +08:00
Code Issues Packages Projects Releases Wiki Activity

[Object Spilling] Initial Iteration of S3 adapter. (#11379)

Browse Source 
* Finished the first iteration.

* Removed unnecessary code.

* Smartopen impl.

* Make sure tests passed.

* Addressed code review.

* Addressed code review.

* Fix issues.

* Fix issues.
This commit is contained in:
SangBin Cho
2020-10-30 14:47:07 -07:00
committed by GitHub
parent 7aade469d0
commit 71c5089854
4 changed files with 229 additions and 44 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/ray/parameter.py
+8
View File
@@ -320,3 +320,11 @@ class RayParams:
if numpy_major <= 1 and numpy_minor < 16:
logger.warning("Using ray with numpy < 1.16.0 will result in slow "
"serialization. Upgrade numpy if using with ray.")
# Make sure object spilling configuration is applicable.
object_spilling_config = self.object_spilling_config or {}
if object_spilling_config:
from ray import external_storage
# Validate external storage usage.
external_storage.setup_external_storage(object_spilling_config)
external_storage.reset_external_storage()