diff --git a/doc/source/advanced.rst b/doc/source/advanced.rst index 742fbfa68..ea431bb24 100644 --- a/doc/source/advanced.rst +++ b/doc/source/advanced.rst @@ -392,3 +392,39 @@ To get information about the current available resource capacity of your cluster .. autofunction:: ray.available_resources :noindex: + +Object Spilling +--------------- + +Ray 1.2.0+ has *beta* support for spilling objects to external storage once the capacity +of the object store is used up. Please file a `GitHub issue `__ +if you encounter any problems with this new feature. Eventually, object spilling will be +enabled by default, but for now you need to enable it manually: + +To enable object spilling to the local filesystem (single node clusters only): + +.. code-block:: python + + ray.init( + _system_config={ + "automatic_object_spilling_enabled": True, + "object_spilling_config": json.dumps( + {"type": "filesystem", "params": {"directory_path": "/tmp/spill"}}, + ) + }, + ) + +To enable object spilling to remote storage (any URI supported by `smart_open `__): + +.. code-block:: python + + ray.init( + _system_config={ + "automatic_object_spilling_enabled": True, + "max_io_workers": 4, # More IO workers for remote storage. + "min_spilling_size": 100 * 1024 * 1024, # Spill at least 100MB at a time. + "object_spilling_config": json.dumps( + {"type": "smart_open", "params": {"uri": "s3:///bucket/path"}}, + ) + }, + ) diff --git a/doc/source/walkthrough.rst b/doc/source/walkthrough.rst index 2a4b91716..8680c1af3 100644 --- a/doc/source/walkthrough.rst +++ b/doc/source/walkthrough.rst @@ -417,6 +417,8 @@ actors. to the object ref returned by the put exists. This only applies to the specific ref returned by put, not refs in general or copies of that refs. +See also: `object spilling `__. + Remote Classes (Actors) -----------------------