XRay Task Forwarding Milestone (#1785)

Summary:
Able to run 1000 tasks with object dependencies on a set of distributed Raylets.

Raylet Changes:

Finalized ClientConnection class.
Task forwarding.
NM-to-NM heartbeats.
NM resource accounting for tasks.
Simple scheduling policy with task forwarding.
Creating and maintaining NM 2 NM long-lived connections and reusing them for task forwarding.
LineageCache Changes:

LineageCache without cleanup of tasks committed by remote nodes.
Lineage cache writeback and cleanup implementation.
ObjectManager Changes:

Object manager event loop/ClientConnection refactor.
Multithreaded object manager (disabled in this PR).
Testing Changes:

Integration tests for task submission on multiple Raylets.
Stress tests for object manager (with GCS and object store integration).


Co-authored-by: Stephanie Wang <swang@cs.berkeley.edu>
Co-authored-by: Alexey Tumanov <atumanov@gmail.com>

src/ray/object_manager/object_store_notification_manager.h

@@ -0,0 +1,72 @@
+#ifndef RAY_OBJECT_MANAGER_OBJECT_STORE_CLIENT_H
+#define RAY_OBJECT_MANAGER_OBJECT_STORE_CLIENT_H
+
+#include <list>
+#include <memory>
+#include <vector>
+
+#include <boost/asio.hpp>
+#include <boost/asio/error.hpp>
+#include <boost/bind.hpp>
+
+#include "plasma/client.h"
+#include "plasma/events.h"
+#include "plasma/plasma.h"
+
+#include "ray/id.h"
+#include "ray/status.h"
+
+#include "ray/object_manager/object_directory.h"
+
+namespace ray {
+
+/// \class ObjectStoreClientPool
+///
+/// Encapsulates notification handling from the object store.
+class ObjectStoreNotificationManager {
+ public:
+  /// Constructor.
+  ///
+  /// \param io_service The asio service to be used.
+  /// \param store_socket_name The store socket to connect to.
+  ObjectStoreNotificationManager(boost::asio::io_service &io_service,
+                                 const std::string &store_socket_name);
+
+  /// Subscribe to notifications of objects added to local store.
+  /// Upon subscribing, the callback will be invoked for all objects that
+  /// already exist in the local store
+  ///
+  /// \param callback A callback expecting an ObjectID.
+  void SubscribeObjAdded(std::function<void(const ray::ObjectID &)> callback);
+
+  /// Subscribe to notifications of objects deleted from local store.
+  ///
+  /// \param callback A callback expecting an ObjectID.
+  void SubscribeObjDeleted(std::function<void(const ray::ObjectID &)> callback);
+
+  /// Terminate this object.
+  void Terminate();
+
+ private:
+  /// Async loop for handling object store notifications.
+  void NotificationWait();
+  void ProcessStoreLength(const boost::system::error_code &error);
+  void ProcessStoreNotification(const boost::system::error_code &error);
+
+  /// Support for rebroadcasting object add/rem events.
+  void ProcessStoreAdd(const ObjectID &object_id);
+  void ProcessStoreRemove(const ObjectID &object_id);
+
+  std::vector<std::function<void(const ray::ObjectID &)>> add_handlers_;
+  std::vector<std::function<void(const ray::ObjectID &)>> rem_handlers_;
+
+  plasma::PlasmaClient store_client_;
+  int c_socket_;
+  int64_t length_;
+  std::vector<uint8_t> notification_;
+  boost::asio::local::stream_protocol::socket socket_;
+};
+
+}  // namespace ray
+
+#endif  // RAY_OBJECT_MANAGER_OBJECT_STORE_CLIENT_H