Enabling the cancellation of non-actor tasks in a worker's queue 2 (#13244)

* wrote code to enable cancellation of queued non-actor tasks

* minor changes

* bug fixes

* added comments

* rev1

* linting

* making ActorSchedulingQueue::CancelTaskIfFound raise a fatal error

* bug fix

* added two unit tests

* linting

* iterating through pending_normal_tasks starting from end

* fixup! iterating through pending_normal_tasks starting from end

* fixup! fixup! iterating through pending_normal_tasks starting from end

* post merge fixes

* added debugging instructions, pulled Accept() out of guarded loop

* removed debugging instructions, linting

* first commit

* lint

* lint

* added hack to avoid race condition in test stress

* moved hack

* fix test cancel

* removed hack (hopefully no longer needed)

* Revert "removed hack (hopefully no longer needed)"

This reverts commit 99d0e7c91539f290700f50aaaed805dcde04a5ee.

* added sleep in mock_worker.cc

* sleep function fixup to work on windows

* sleep in test_fast both for force=true and force=false

* linting

Co-authored-by: Ian <ian.rodney@gmail.com>

python/ray/tests/test_cancel.py

@@ -175,6 +175,8 @@ def test_stress(shutdown_only, use_force):
     sleep_or_no = [random.randint(0, 1) for _ in range(100)]
     tasks = [infinite_sleep.remote(i) for i in sleep_or_no]
     cancelled = set()
+
+    # Randomly kill queued tasks (infinitely sleeping or not).
     for t in tasks:
         if random.random() > 0.5:
             ray.cancel(t, force=use_force)
@@ -186,10 +188,13 @@ def test_stress(shutdown_only, use_force):
     for done in cancelled:
         with pytest.raises(valid_exceptions(use_force)):
             ray.get(done, timeout=120)
+
+    # Kill all infinitely sleeping tasks (queued or not).
     for indx, t in enumerate(tasks):
         if sleep_or_no[indx]:
             ray.cancel(t, force=use_force)
             cancelled.add(t)
+    for indx, t in enumerate(tasks):
         if t in cancelled:
             with pytest.raises(valid_exceptions(use_force)):
                 ray.get(t, timeout=120)
@@ -213,8 +218,8 @@ def test_fast(shutdown_only, use_force):
         # between a worker receiving a task and the worker executing
         # that task (specifically the python execution), Cancellation
         # can fail.
-        if not use_force:
-            time.sleep(0.1)
+
+        time.sleep(0.1)
         ray.cancel(x, force=use_force)
         ids.append(x)