Use flatbuffers for some messages from Redis. (#341)

* Compile the Ray redis module with C++.

* Redo parsing of object table notifications with flatbuffers.

* Update redis module python tests.

* Redo parsing of task table notifications with flatbuffers.

* Fix linting.

* Redo parsing of db client notifications with flatbuffers.

* Redo publishing of local scheduler heartbeats with flatbuffers.

* Fix linting.

* Remove usage of fixed-width formatting of scheduling state in channel name.

* Reply with flatbuffer object to task table queries, also simplify redis string to flatbuffer string conversion.

* Fix linting and tests.

* fix

* cleanup

* simplify logic in ReplyWithTask
This commit is contained in:
Robert Nishihara
2017-03-10 18:35:25 -08:00
committed by Philipp Moritz
parent 555dcf35a2
commit 53dffe0bf2
17 changed files with 379 additions and 385 deletions
+10 -3
View File
@@ -8,6 +8,9 @@ import numpy as np
import time
import redis
# Import flatbuffer bindings.
from ray.core.generated.TaskReply import TaskReply
class TaskTests(unittest.TestCase):
def testSubmittingTasks(self):
@@ -164,9 +167,13 @@ class ReconstructionTests(unittest.TestCase):
r = redis.StrictRedis(port=self.redis_port)
task_ids = r.keys("TT:*")
task_ids = [task_id[3:] for task_id in task_ids]
node_ids = [r.execute_command("ray.task_table_get", task_id)[1] for task_id
in task_ids]
self.assertEqual(len(set(node_ids)), self.num_local_schedulers)
local_scheduler_ids = []
for task_id in task_ids:
message = r.execute_command("ray.task_table_get", task_id)
task_reply_object = TaskReply.GetRootAsTaskReply(message, 0)
local_scheduler_ids.append(task_reply_object.LocalSchedulerId())
self.assertEqual(len(set(local_scheduler_ids)), self.num_local_schedulers)
# Clean up the Ray cluster.
ray.worker.cleanup()