wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-06-28 11:53:32 +08:00
Code Issues Packages Projects Releases Wiki Activity

Prevent hasher from running out of memory on large files (#2451)

Browse Source 
* Prevent hasher from running out of memory on large files

* dump out keys

* only print if failed

* remove debugging

* Fix lint error. Reverse adding newline.
This commit is contained in:
Steve Severance
2018-07-28 23:29:09 -07:00
committed by Eric Liang
parent 80db69d245
commit f1b4ea69a3
1 changed files with 6 additions and 1 deletions
Download Patch File Download Diff File Expand all files Collapse all files
python/ray/autoscaler/autoscaler.py
+6 -1
View File
@@ -653,7 +653,12 @@ def hash_runtime_conf(file_mounts, extra_objs):
for name in filenames:
hasher.update(name.encode("utf-8"))
with open(os.path.join(dirpath, name), "rb") as f:
hasher.update(binascii.hexlify(f.read()))
if os.path.getsize(os.path.join(dirpath,
name)) < 1000000000:
hasher.update(binascii.hexlify(f.read()))
else:
for chunk in iter(lambda: f.read(8192), b''):
hasher.update(binascii.hexlify(chunk))
else:
with open(path, "rb") as f:
hasher.update(binascii.hexlify(f.read()))