Add "beta" documentation for enabling object spilling manually (#13047)

This commit is contained in:
Eric Liang
2020-12-22 16:08:41 -08:00
committed by GitHub
parent a79c9fcac3
commit 81d3cbaa77
2 changed files with 38 additions and 0 deletions
+36
View File
@@ -392,3 +392,39 @@ To get information about the current available resource capacity of your cluster
.. autofunction:: ray.available_resources
:noindex:
Object Spilling
---------------
Ray 1.2.0+ has *beta* support for spilling objects to external storage once the capacity
of the object store is used up. Please file a `GitHub issue <https://github.com/ray-project/ray/issues/>`__
if you encounter any problems with this new feature. Eventually, object spilling will be
enabled by default, but for now you need to enable it manually:
To enable object spilling to the local filesystem (single node clusters only):
.. code-block:: python
ray.init(
_system_config={
"automatic_object_spilling_enabled": True,
"object_spilling_config": json.dumps(
{"type": "filesystem", "params": {"directory_path": "/tmp/spill"}},
)
},
)
To enable object spilling to remote storage (any URI supported by `smart_open <https://pypi.org/project/smart-open/>`__):
.. code-block:: python
ray.init(
_system_config={
"automatic_object_spilling_enabled": True,
"max_io_workers": 4, # More IO workers for remote storage.
"min_spilling_size": 100 * 1024 * 1024, # Spill at least 100MB at a time.
"object_spilling_config": json.dumps(
{"type": "smart_open", "params": {"uri": "s3:///bucket/path"}},
)
},
)
+2
View File
@@ -417,6 +417,8 @@ actors.
to the object ref returned by the put exists. This only applies to the specific
ref returned by put, not refs in general or copies of that refs.
See also: `object spilling <advanced.html#object-spilling>`__.
Remote Classes (Actors)
-----------------------