mirror of
https://github.com/wassname/ray.git
synced 2026-08-11 11:24:51 +08:00
[Dashboard] Refresh documentation 1.0.0 (#10684)
This commit is contained in:
Max Fitton
@@ -4,6 +4,7 @@ Ray's built-in dashboard provides metrics, charts, and other features that help
|
||||
Ray users to understand Ray clusters and libraries.
|
||||
|
||||
The dashboard lets you:
|
||||
|
||||
- View cluster metrics.
|
||||
- See errors and exceptions at a glance.
|
||||
- View logs across many machines in a single pane.
|
||||
@@ -13,9 +14,6 @@ The dashboard lets you:
|
||||
- See Tune jobs and trial information.
|
||||
- Detect cluster anomalies and debug them.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Dashboard-overview.png
|
||||
:align: center
|
||||
|
||||
Getting Started
|
||||
---------------
|
||||
You can access the dashboard through its default URL, **localhost:8265**.
|
||||
@@ -34,47 +32,62 @@ The dashboard is also available :ref:`when using the cluster launcher <monitor-c
|
||||
|
||||
Views
|
||||
-----
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/dashboard-component-view.png
|
||||
:align: center
|
||||
|
||||
Machine View
|
||||
~~~~~~~~~~~~
|
||||
The machine view shows you:
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/machine-view-overview.png
|
||||
:align: center
|
||||
|
||||
- System resource usage for each machine and worker such as RAM, CPU, disk, and network usage information.
|
||||
- Logs and error messages for each machine and worker.
|
||||
- Actors or tasks assigned to each worker process.
|
||||
The machine view lets you see resource utilization information on a per-node and per-worker basis. This also shows the assignment of GPU resources to specific actors or tasks.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Machine-view-basic.png
|
||||
In addition, the machine view lets you see **logs** and **error messages**. You can see messages for the whole cluster, or drill down into a specific node or worker.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/machine-view-logs.png
|
||||
:align: center
|
||||
|
||||
Finally, you can see the task that each worker is currently performing.
|
||||
|
||||
|
||||
Logical View
|
||||
~~~~~~~~~~~~
|
||||
The logical view shows you:
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-overview.png
|
||||
:align: center
|
||||
|
||||
- Created and killed actors.
|
||||
- State of actors (e.g. Alive, Dead, Pending Creation). Learn more about actor states at
|
||||
- Actor statistics such as number of executed tasks, pending tasks, and memory usage.
|
||||
- Actor hierarchy.
|
||||
The logical view lets you monitor the actors running on your Ray cluster. For each actor class defined in your program, the logical view allows you to see how many of that class are running, how many tasks they've executed, and more.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Logical-view-basic.png
|
||||
In addition, it will warn you if you have an actor that cannot be created because your cluster has insufficient resources to satisfy its requirements.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-warning.png
|
||||
:align: center
|
||||
|
||||
You can expand the panel for a class to see more detailed information about individual actors. You can profile these, view their logs, and see information about their arguments.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-expanded-actors.png
|
||||
:align: center
|
||||
|
||||
Memory View
|
||||
~~~~~~~~~~~~
|
||||
The memory view shows you:
|
||||
The memory view lets you see information about the data stored in the Ray Object store. It is very useful if you have encountered an ``ObjectStoreFullError``. A number of things are stored in the object store, including:
|
||||
|
||||
- The state of Ray objects, including their size, reference type, and call site.
|
||||
- The aggregate amount of memory being used by various groups, such as line of code, or the node.
|
||||
1. Actor References
|
||||
2. Objects returned from a task or actor method
|
||||
3. Objects that are passed into an actor as an argument
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/images/master/docs/dashboard/Memory-view-basic.png
|
||||
You can group the memory view by node, or by the stack trace that created the memory being used. The latter is particularly helpful for tracking down the **line of code where a memory leak occurs**. See more below in `Debugging ObjectStoreFullError and Memory Leaks`_.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/memory-view-stack-trace.png
|
||||
:align: center
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/memory-view-expanded.png
|
||||
:align: center
|
||||
|
||||
.. note::
|
||||
This view does **not** show information about **heap memory usage**. That means if your actor or task is allocating too much memory locally (not in the Ray object store a.k.a Plasma), this view will not help you find it.
|
||||
|
||||
Ray Config
|
||||
~~~~~~~~~~
|
||||
|
||||
The ray config tab shows you the current autoscaler configuration.
|
||||
The ray config tab shows you the current cluster launcher configuration if you're using the ray cluster launcher. The cluster launcher was formerly known as the autoscaler.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Ray-config-basic.png
|
||||
:align: center
|
||||
@@ -111,37 +124,15 @@ are shown in yellow.
|
||||
|
||||
Below is an example.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
|
||||
ray.init(num_gpus=2)
|
||||
|
||||
@ray.remote(num_gpus=1)
|
||||
class Actor1:
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
@ray.remote(num_gpus=4)
|
||||
class Actor2:
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
actor1_list = [Actor1.remote() for _ in range(4)]
|
||||
actor2 = Actor2.remote()
|
||||
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/dashboard-pending-infeasible-actors.png
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-overview.png
|
||||
:align: center
|
||||
|
||||
This cluster has two GPUs, and so it only has room to create two copies of ``Actor1``.
|
||||
As a result, the rest of ``Actor1`` will be pending.
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-warning.png
|
||||
:align: center
|
||||
|
||||
You can also see it is infeasible to create ``Actor2`` because it requires 4 GPUs which
|
||||
is bigger than the total gpus available in this cluster (2 GPUs).
|
||||
|
||||
Debugging ObjectStoreFullError and Memory Leak
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
Debugging ObjectStoreFullError and Memory Leaks
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
You can view information for Ray objects in the memory tab. It is useful to debug memory leaks, especially `ObjectStoreFullError`.
|
||||
|
||||
One common cause of these memory errors is that there are objects which never go out of scope. In order to find these, you can go to the Memory View, then select to "Group By Stack Trace." This groups memory entries by their stack traces up to three frames deep. If you see a group which is growing without bound, you might want to examine that line of code to see if you intend to keep that reference around.
|
||||
@@ -246,12 +237,14 @@ Note that number of workers can exceed number of cores.
|
||||
**Errors**: Error messages at each node and worker. You can see error messages by clicking it.
|
||||
|
||||
|
||||
Logical View (Experimental)
|
||||
~~~~~~~~~~~~~~~~~~~~~~~~~~~
|
||||
**Actor Titles**: Name of an actor and its arguments.
|
||||
Logical View
|
||||
~~~~~~~~~~~~
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/logical-view-expanded-actors.png
|
||||
:align: center
|
||||
|
||||
**Title**: Name of an actor and its arguments.
|
||||
|
||||
**State**: State of an actor.
|
||||
|
||||
- Alive
|
||||
- Restarting
|
||||
- Dead
|
||||
@@ -259,8 +252,6 @@ Logical View (Experimental)
|
||||
- Pending Creation
|
||||
- Dependencies Unready (waiting for one or more of its arguments to be ready)
|
||||
|
||||
**Number of Pending Tasks**: The number of method calls for this actor that are still awaiting execution.
|
||||
|
||||
**Number of Excuted Tasks**: A number of completed method calls for this actor.
|
||||
|
||||
**Number of ObjectRefs In Scope**: The number of object refs in scope for this actor, which correspond to objects in the Ray object store. object refs
|
||||
@@ -277,45 +268,6 @@ objects are staying in local memory.
|
||||
**profile**: A button to run profiling. We currently support profiling for 10s,
|
||||
30s and 60s. It requires passwordless ``sudo``. The result of profiling is a py-spy html output displaying how much CPU time the actor spent in various methods.
|
||||
|
||||
**Infeasible Actor Creation**: Actor creation is infeasible when an actor
|
||||
requires more resources than a Ray cluster can provide, for example an actor that requires a GPU on a cluster that has none. The actor's state is marked "Infeasible" and highlighted in red.
|
||||
|
||||
**Pending Actor Creation**: Actor creation is pending when there are no
|
||||
available resources for this actor because they are already taken by other
|
||||
tasks and actors. This is depicted as a yellow colored actor.
|
||||
|
||||
**Actor Hierarchy**: The logical view renders actor information in a tree format.
|
||||
|
||||
To illustrate this, in the code block below, the ``Parent`` actor creates
|
||||
two ``Child`` actors and each ``Child`` actor creates one ``GrandChild`` actor.
|
||||
This relationship is visible in the dashboard *Logical View* tab.
|
||||
|
||||
.. code-block:: python
|
||||
|
||||
import ray
|
||||
ray.init()
|
||||
|
||||
@ray.remote
|
||||
class Grandchild:
|
||||
def __init__(self):
|
||||
pass
|
||||
|
||||
@ray.remote
|
||||
class Child:
|
||||
def __init__(self):
|
||||
self.grandchild_handle = Grandchild.remote()
|
||||
|
||||
@ray.remote
|
||||
class Parent:
|
||||
def __init__(self):
|
||||
self.children_handles = [Child.remote() for _ in range(2)]
|
||||
|
||||
parent_handle = Parent.remote()
|
||||
|
||||
You can see that the dashboard shows the parent/child relationship as expected.
|
||||
|
||||
.. image:: https://raw.githubusercontent.com/ray-project/Images/master/docs/dashboard/Logical-view-basic.png
|
||||
:align: center
|
||||
|
||||
Memory
|
||||
~~~~~~
|
||||
@@ -337,7 +289,6 @@ Memory
|
||||
|
||||
Ray Config
|
||||
~~~~~~~~~~~~
|
||||
|
||||
If you are using the cluster launcher, this Configuration defined at ``cluster.yaml`` is shown.
|
||||
See `Cluster.yaml reference <https://github.com/ray-project/ray/blob/master/python/ray/autoscaler/aws/example-full.yaml>`_ for more details.
|
||||
|
||||
|
||||
Reference in New Issue
Block a user