wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-07-12 08:03:12 +08:00
Code Issues Packages Projects Releases Wiki Activity

Improved Pickle5 pickling (#5841)

Browse Source 
* object copy optimization

* see if we can reuse the Arrow parallel_memcopy

* remove unused function

* restore the original code, since later experiments show that it has little impact on performance.

* lint
This commit is contained in:
Si-Yuan
2019-10-03 15:14:32 -07:00
committed by GitHub
parent fa1214c44a
commit 3a42780cb8
2 changed files with 5 additions and 43 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/ray/worker.py
+2 -2
View File
@@ -434,9 +434,9 @@ class Worker(object):
value, object_id, memcopy_threads=self.memcopy_threads)
else:
writer = Pickle5Writer()
meta = pickle.dumps(
inband = pickle.dumps(
value, protocol=5, buffer_callback=writer.buffer_callback)
self.core_worker.put_pickle5_buffers(object_id, meta, writer,
self.core_worker.put_pickle5_buffers(object_id, inband, writer,
self.memcopy_threads)
except pyarrow.plasma.PlasmaObjectExists:
# The object already exists in the object store, so there is no