[Dashboard] Memory View Group by Stack Trace and UI Overhaul (#10227)

This commit is contained in:
Max Fitton
2020-08-24 14:54:42 -05:00
committed by GitHub
parent c8c4832794
commit 832f5cdccb
14 changed files with 463 additions and 414 deletions
+11 -6
View File
@@ -3,10 +3,12 @@ Ray Dashboard
Ray's built-in dashboard provides metrics, charts, and other features that help
Ray users to understand Ray clusters and libraries.
Through the dashboard, you can
The dashboard lets you:
- View cluster metrics.
- Visualize the actor relationships and statistics.
- See errors and exceptions at a glance.
- View logs across many machines in a single pane.
- Understand Ray memory utilization and debug memory errors.
- See per-actor resource usage, executed tasks, logs, and more.
- Kill actors and profile your Ray jobs.
- See Tune jobs and trial information.
- Detect cluster anomalies and debug them.
@@ -64,7 +66,7 @@ Memory View
The memory view shows you:
- The state of Ray objects, including their size, reference type, and call site.
- A summary of reference types and object sizes in use.
- The aggregate amount of memory being used by various groups, such as line of code, or the node.
.. image:: https://raw.githubusercontent.com/ray-project/images/master/docs/dashboard/Memory-view-basic.png
:align: center
@@ -141,6 +143,9 @@ is bigger than the total gpus available in this cluster (2 GPUs).
Debugging ObjectStoreFullError and Memory Leak
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
You can view information for Ray objects in the memory tab. It is useful to debug memory leaks, especially `ObjectStoreFullError`.
One common cause of these memory errors is that there are objects which never go out of scope. In order to find these, you can go to the Memory View, then select to "Group By Stack Trace." This groups memory entries by their stack traces up to three frames deep. If you see a group which is growing without bound, you might want to examine that line of code to see if you intend to keep that reference around.
Note that this is the same information as displayed in the `ray memory command <https://docs.ray.io/en/latest/memory-management.html#debugging-using-ray-memory>`_. For details about the information contained in the table, please see the `ray memory` documentation.
Inspect Memory Usage
@@ -318,7 +323,7 @@ Memory
**IP Address**: Node IP Address where a Ray object is pinned.
**Pid**: ID of a process where a Ray object is being used.
**PID**: ID of a process where a Ray object is being used.
**Type**: Type of a process. It is either a driver or worker.
@@ -328,7 +333,7 @@ Memory
**Reference Type**: Reference types of Ray objects. Checkout the `ray memory command <https://docs.ray.io/en/latest/memory-management.html#debugging-using-ray-memory>`_ to learn each reference type.
**Call Site**: Call site where this Ray object is referenced.
**Call Site**: Call site where this Ray object is referenced, up to three stack frames deep.
Ray Config
~~~~~~~~~~~~