[Documentation] local_mode doc updates and actor / worker explanation from Slack (#10748)

* wip

* Update local mode docs in all locations

* Update doc/source/actors.rst

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

* Update doc/source/actors.rst

Co-authored-by: Richard Liaw <rliaw@berkeley.edu>

* Change duplicated text to links to a subtitle for local_mode

* change a reference to be explicit

* Apply suggestions from code review

Co-authored-by: Max Fitton <max@semprehealth.com>
Co-authored-by: Richard Liaw <rliaw@berkeley.edu>
Resolved Conflicts:
	doc/source/actors.rst

doc/examples/testing-tips.rst

@@ -20,6 +20,8 @@ This may easily result in tests exhibiting unexpected, flaky, or faulty behavior
 To overcome this, you should override the detected resources by setting them in ``ray.init`` like: ``ray.init(num_cpus=2)``
 
 
+.. _local-mode-tips:
+
 Tip 2: Use ``ray.init(local_mode=True)`` if possible
 ----------------------------------------------------
 
@@ -32,6 +34,10 @@ However, there are some caveats with using this. You should not do this if:
 1. If your application depends on setting environment variables per process
 2. If your application has recursive actor calls
 3. If your remote actor/task sets any sort of process-level global variables
+4. If you use are using async actors
+
+Also note, if you are using GPUs, you must set the ``CUDA_VISIBLE_DEVICES`` environment
+variable to a comma separated list of your GPU Device IDs.
 
 
 Tip 3: Sharing the ray cluster across tests if possible
@@ -137,4 +143,4 @@ See the `Cluster Util for more details <https://github.com/ray-project/ray/blob/
 Tip 5: Be careful when running tests in parallel
 ------------------------------------------------
 
-Since Ray starts a variety of services, it is easy to trigger timeouts if too many services are started at once. Therefore, when using tools such as `pytest xdist <https://pypi.org/project/pytest-xdist/>`_ that run multiple tests in parallel, one should keep in mind that this may introduce flakiness into the test environment.
+Since Ray starts a variety of services, it is easy to trigger timeouts if too many services are started at once. Therefore, when using tools such as `pytest xdist <https://pypi.org/project/pytest-xdist/>`_ that run multiple tests in parallel, one should keep in mind that this may introduce flakiness into the test environment.

doc/source/actors.rst

@@ -376,6 +376,9 @@ This allows you to retrieve the actor from any job in the Ray cluster.
 This can be useful if you cannot directly
 pass the actor handle to the task that needs it, or if you are trying to
 access an actor launched by another driver.
+Note that the actor will still be garbage-collected if no handles to it
+exist. See :ref:`actor-lifetimes` for more details.
+
 
 .. tabs::
 
@@ -415,6 +418,7 @@ access an actor launched by another driver.
       Optional<ActorHandle<Counter>> counter = Ray.getActor("some_name_in_job");
       Assert.assertTrue(counter.isPresent());
 
+.. _actor-lifetimes:
 
 Actor Lifetimes
 ---------------
@@ -469,3 +473,43 @@ Actor Pool
   .. group-tab:: Java
 
     Actor pool hasn't been implemented in Java yet.
+
+
+Actors, Workers and Resources
+-----------------------------
+What's the difference between a worker and an actor?
+
+Each "Ray worker" is a python process. 
+
+Workers are treated differently for tasks and actors. Any "Ray worker" is either 1. used to execute multiple Ray tasks or 2. is started as a dedicated Ray actor.
+
+ * Tasks: When Ray starts on a machine, a number of Ray workers will be started automatically (1 per CPU by default). They will be used to execute tasks (like a process pool). If you execute 8 tasks with `num_cpus=2`, and total number of CPUs is 16 (`ray.cluster_resources()["CPU"] == 16`), you will end up with 8 of your 16 workers idling.
+
+ * Actor: A Ray Actor is also a "Ray worker" but is instantiated at runtime (upon `actor_cls.remote()`). All of its methods will run on the same process, using the same resources (designated when defining the Actor). Note that unlike tasks, the python processes that runs Ray Actors are not reused and will be terminated when the Actor is deleted.
+
+To maximally utilize your resources, you want to maximize the time that
+your workers are working. You also want to allocate enough cluster resources 
+so that both all of your needed actors can run and any other tasks you
+define can run. This also implies that tasks are scheduled more flexibly,
+and that if you don't need the stateful part of an actor, you're mostly
+better off using tasks.
+
+
+Concurrency within an actor
+---------------------------
+
+.. tabs::
+  .. group-tab:: Python
+
+    Within a single actor process, it is possible to execute concurrent threads.
+
+    Ray offers two types of concurrency within an actor:
+
+     * :ref:`async execution <async-actors>`
+     * :ref:`threading <threaded-actors>`
+
+    See the above links for more details.
+
+  .. group-tab:: Java
+
+    Actor-level concurrency hasn't been implemented in Java yet.

doc/source/starting-ray.rst

@@ -217,7 +217,7 @@ By default, Ray will parallelize its workload and run tasks on multiple processe
         -Dray.local-mode=true \
         <classname> <args>
 
-Note that some behavior such as setting global process variables may not work as expected.
+Note that there are some known issues with local mode. Please read :ref:`these tips <local-mode-tips>` for more information.
 
 .. note:: If you just want to run your Java code in local mode, you can run it without Ray or even Python installed.
 

doc/source/tune/user-guide.rst

@@ -521,6 +521,8 @@ By default, Tune will run hyperparameter evaluations on multiple processes. Howe
 
 Local mode with multiple configuration evaluations will interleave computation, so it is most naturally used when running a single configuration evaluation.
 
+Note that ``local_mode`` has some known issues, so please read :ref:`these tips <local-mode-tips>` for more info.
+
 Stopping after the first failure
 --------------------------------