Stream logs to driver by default. (#3892)

* Stream logs to driver by default.

* Fix from rebase

* Redirect raylet output independently of worker output.

* Fix.

* Create redis client with services.create_redis_client.

* Suppress Redis connection error at exit.

* Remove thread_safe_client from redis.

* Shutdown driver threads in ray.shutdown().

* Add warning for too many log messages.

* Only stop threads if worker is connected.

* Only stop threads if they exist.

* Remove unnecessary try/excepts.

* Fix

* Only add new logging handler once.

* Increase timeout.

* Fix tempfile test.

* Fix logging in cluster_utils.

* Revert "Increase timeout."

This reverts commit b3846b89040bcd8e583b2e18cb513cb040e71d95.

* Retry longer when connecting to plasma store from node manager and object manager.

* Close pubsub channels to avoid leaking file descriptors.

* Limit log monitor open files to 200.

* Increase plasma connect retries.

* Add comment.

python/ray/utils.py

@@ -276,21 +276,11 @@ def setup_logger(logging_level, logging_format):
         logging_level = logging.getLevelName(logging_level.upper())
     logger.setLevel(logging_level)
     global _default_handler
-    _default_handler = logging.StreamHandler()
-    _default_handler.setFormatter(logging.Formatter(logging_format))
-    logger.addHandler(_default_handler)
-    logger.propagate = False
-
-
-def try_update_handler(new_stream):
-    global _default_handler
-    logger = logging.getLogger("ray")
-    if _default_handler:
-        new_handler = logging.StreamHandler(stream=new_stream)
-        new_handler.setFormatter(_default_handler.formatter)
-        _default_handler.close()
-        _default_handler = new_handler
+    if _default_handler is None:
+        _default_handler = logging.StreamHandler()
         logger.addHandler(_default_handler)
+    _default_handler.setFormatter(logging.Formatter(logging_format))
+    logger.propagate = False
 
 
 # This function is copied and modified from