[Core] Log output from different jobs to different drivers. (#8885)

* .

* .

* Correct now

* No interactivity errors

* format

* Filtering

* lint

* .

* No more filtering

* Removed interactivity

* .

* .

* .

* .

* .

* .

* Redirection works

* formatting

* something broken?

* .

* Works

* formatting

* redirect output

* formatting

* formatting

* Fix file descriptor leakage

* format

* .

* .

* .

* .

* .

* Refactor

* .

* Only run on job switch

* .

* cleanup

* .

* ...

* Review

* .

* .

* .

* .

* whoops

* .

* Should fix bug

* .

* .

* addressed comments

* formatting

* formatting

* Fix typo

* .

* .

* .

* .

Co-authored-by: Ubuntu <ubuntu@ip-172-31-14-33.us-west-2.compute.internal>

python/ray/utils.py

@@ -390,6 +390,36 @@ def setup_logger(logging_level, logging_format):
     logger.propagate = False
 
 
+def open_log(path, **kwargs):
+    kwargs.setdefault("buffering", 1)
+    kwargs.setdefault("mode", "a")
+    kwargs.setdefault("encoding", "utf-8")
+    return open(path, **kwargs)
+
+
+def create_and_init_new_worker_log(path, worker_pid):
+    """
+    Opens a path (or creates if necessary) for a log.
+
+    Args:
+        path (str): The name/path of the file to be opened.
+
+    Returns:
+        A file-like object which can be written to.
+    """
+    # TODO (Alex): We should eventually be able to replace this with
+    # named-pipes.
+    f = open_log(path)
+    # Check to see if we're creating this file. No one else should ever write
+    # to this file, so we don't have to worry about TOCTOU.
+    if f.tell() == 0:
+        # This should always be the first message to appear in the worker's
+        # stdout and stderr log files. The string "Ray worker pid:" is
+        # parsed in the log monitor process.
+        print("Ray worker pid: {}".format(worker_pid), file=f)
+    return f
+
+
 def get_system_memory():
     """Return the total amount of system memory in bytes.