Stephanie Wang d7eef808b8 [core] Reconstruction for lost plasma objects (#7733) ...

* Add a lineage_ref_count to References

* Refactor TaskManager to store TaskEntry as a struct

* Refactor to fix deadlock between TaskManager and ReferenceCounter
Add references to task specs

* Pin TaskEntries and References in the lineage of any ObjectIDs in scope

* Fix deadlock, convert num_plasma_returns to a set of object IDs

* fix unit tests

* Feature flag

* Do not release lineage for objects that were promoted to plasma

* fix build

* fix build

* Remove num executions

* Remove num executions

* Add pinned locations to ReferenceCounter, empty handler for node death

* Fix num returns for actor tasks, fix Put return value

* Add regression test

* Clear pinned locations and callbacks on node removal

* Clear pinned locations and callbacks on node removal

* Simplify num return values

* Remove unused

* doc

* tmp

* Set num returns

* Move lineage pinning flag to ReferenceCounter

* comments

* Recover from plasma failures by pinning a new copy

* Basic object reconstruction, no concurrent reqs yet

* reconstruction test suite and a few fixes:
- fix for disabling lineage
- fix for updating submitted task refs

* Handle concurrent attempts to recover the same object

* Fix deadlock in DrainAndShutdown

* Revert "[core] Revert lineage pinning (#7499) (#7692)"

This reverts commit ba86a02b37.

* debug rllib

* debug rllib

* turn on all rllib tests again

* debug rllib

* Fix drain bug, check number of pending tasks

* revert rllib debug

* remove todo

* Trigger rllib tests

* revert rllib debug commit

* Split out logic into ObjectRecoveryManager

* Fix python tests

* Refactor to remove dependency on gcs client

* Unit tests

* Move pinned at node ID to direct memory store

* Unit test fixes and lint

* simplify and more tests

* Add ResubmitTask test for TaskManager

* Doc

* fix build

* comments

* Fix

* debug

* Update

* fix

* Fix

* Fix bad status handling, unit test

* Fix build

2020-04-11 16:52:57 -07:00

..

autoscaler

[autoscaler] Allowing users to provide extra configs for AWS (#7844)

2020-04-04 18:36:51 -07:00

cloudpickle

Python 3.8 compatibility (#7754)

2020-04-01 10:03:23 -07:00

core

Define common data structures with protobuf. (#5121)

2019-07-08 22:41:37 +08:00

dashboard

[Dashboard] Configure Subset of Parameters/Metrics and show Err… (#7726)

2020-04-10 13:27:52 -07:00

dataframe

Dataframe deprecation (#2353)

2018-07-06 00:16:22 -07:00

experimental

Cleaning up remaining Local Mode Code (#7865)

2020-04-03 19:54:15 -05:00

includes

Cross language serialization for primitive types (#7711)

2020-04-08 21:10:57 +08:00

internal

Cleaning up remaining Local Mode Code (#7865)

2020-04-03 19:54:15 -05:00

projects

Change /tmp to platform-specific temporary directory (#7529)

2020-03-16 18:10:14 -07:00

pyarrow_files

Package pyarrow along with ray. (#822)

2017-08-07 21:17:28 -07:00

scripts

--redis-address -> --address (#7760)

2020-03-26 13:52:39 -07:00

serve

[serve] Push requests to workers instead of polling via dequeue_request (#7965)

2020-04-10 14:47:03 -05:00

tests

[core] Reconstruction for lost plasma objects (#7733)

2020-04-11 16:52:57 -07:00

tune

[tune] Ensure Cleanup (#7967)

2020-04-11 16:28:03 -07:00

util

[sgd] Semantic Segmentation Example (#7825)

2020-04-10 20:35:45 -07:00

workers

Deprecate use_pickle flag (#7474)

2020-03-09 16:03:56 -07:00

__init__.py

Remove ray.tasks() from API. (#7807)

2020-04-01 10:10:40 -05:00

_raylet.pxd

Support multiple core workers in one process (#7623)

2020-04-07 11:01:47 +08:00

_raylet.pyx

Change Proctitle to IDLE after an Error (#7863)

2020-04-08 11:33:43 -07:00

actor.py

[serve] Make HTTP proxy fault tolerant (#7936)

2020-04-09 17:07:22 -05:00

async_compat.py

[Serve] Async Router (#6873)

2020-01-30 09:34:47 -08:00

cluster_utils.py

[core] Option to fallback to LRU on OutOfMemory (#7410)

2020-03-14 11:28:43 -07:00

cross_language.py

Remove instances of 'raise Exception' (#7523)

2020-03-10 17:51:22 -07:00

exceptions.py

Remove unnecessary handling of setproctitle and psutil. (#7702)

2020-03-22 22:06:42 -07:00

function_manager.py

Moving Local Mode to C++ (#7670)

2020-04-01 15:50:57 -05:00

gcs_utils.py

Remove future imports (#6724)

2020-01-09 00:15:48 -08:00

import_thread.py

Remove instances of 'raise Exception' (#7523)

2020-03-10 17:51:22 -07:00

log_monitor.py

Remove future imports (#6724)

2020-01-09 00:15:48 -08:00

memory_monitor.py

First pass at ray memory command for memory debugging (#7589)

2020-03-17 20:45:07 -07:00

monitor.py

Remove ray.tasks() from API. (#7807)

2020-04-01 10:10:40 -05:00

node.py

Log to terminal if glog is also doing so (#7868)

2020-04-10 18:41:21 -05:00

parameter.py

Moving Local Mode to C++ (#7670)

2020-04-01 15:50:57 -05:00

profiling.py

Remove future imports (#6724)

2020-01-09 00:15:48 -08:00

ray_cluster_perf.py

Move more unit tests to bazel (#6250)

2019-11-24 11:43:34 -08:00

ray_constants.py

Cross language serialization for primitive types (#7711)

2020-04-08 21:10:57 +08:00

ray_perf.py

Remove (object) from class declarations. (#6658)

2020-01-02 17:42:13 -08:00

ray_process_reaper.py

Windows bug fixes (#7740)

2020-03-30 20:39:23 -05:00

remote_function.py

Moving Local Mode to C++ (#7670)

2020-04-01 15:50:57 -05:00

reporter.py

Windows bug fixes (#7740)

2020-03-30 20:39:23 -05:00

resource_spec.py

Remove instances of 'raise Exception' (#7523)

2020-03-10 17:51:22 -07:00

rllib

[rllib] Try moving RLlib to top level dir (#5324)

2019-08-05 23:25:49 -07:00

serialization.py

Cross language serialization for primitive types (#7711)

2020-04-08 21:10:57 +08:00

services.py

Log to terminal if glog is also doing so (#7868)

2020-04-10 18:41:21 -05:00

setup-dev.py

Add Dashboard and Util to setup-dev (#7321)

2020-02-25 15:25:09 -08:00

signature.py

Remove dependence on funcsigs. (#7701)

2020-03-22 21:37:24 -07:00

state.py

Remove ray.tasks() from API. (#7807)

2020-04-01 10:10:40 -05:00

streaming

[Streaming] Streaming data transfer and python integration (#6185)

2019-12-10 20:33:24 +08:00

test_utils.py

fix test_worker_stats (#7655)

2020-03-20 14:53:40 +08:00

utils.py

Windows bug fixes (#7740)

2020-03-30 20:39:23 -05:00

worker.py

Change Proctitle to IDLE after an Error (#7863)

2020-04-08 11:33:43 -07:00