wassname/ray
1
0
Fork 0
mirror of https://github.com/wassname/ray.git synced 2026-08-11 11:24:51 +08:00
Code Issues Packages Projects Releases Wiki Activity

Contextual Bandit algorithms (WIP) (#7642)

Browse Source
This commit is contained in:
Saurabh Gupta
2020-03-26 13:41:16 -07:00
committed by GitHub
parent c1b05b720d
commit 6ddf84b019
19 changed files with 1174 additions and 2 deletions
Download Patch File Download Diff File Expand all files Collapse all files
doc/source/rllib-algorithms.rst
+62 -1
View File
@@ -465,8 +465,69 @@ Tuned examples: `CartPole-v0 <https://github.com/ray-project/ray/blob/master/rll
:start-after: __sphinx_doc_begin__
:end-before: __sphinx_doc_end__
Contextual Bandits (contrib/bandits)
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The Multi-armed bandit (MAB) problem provides a simplified RL setting that
involves learning to act under one situation only, i.e. the state is fixed.
Contextual bandit is extension of the MAB problem, where at each
round the agent has access not only to a set of bandit arms/actions but also
to a context (state) associated with this iteration. The context changes
with each iteration, but, is not affected by the action that the agent takes.
The objective of the agent is to maximize the cumulative rewards, by
collecting enough information about how the context and the rewards of the
arms are related to each other. The agent does this by balancing the
trade-off between exploration and exploitation.
Contextual bandit algorithms typically consist of an action-value model (Q
model) and an exploration strategy (e-greedy, UCB, Thompson Sampling etc.)
RLlib supports the following online contextual bandit algorithms,
named after the exploration strategies that they employ:
LinUCB (Upper Confidence Bound)
-------------------------------
|pytorch|
`[paper] <http://rob.schapire.net/papers/www10.pdf>`__ `[implementation]
<https://github.com/ray-project/ray/blob/master/rllib/contrib/bandits/agents/lin_ucb.py>`__
LinUCB assumes a linear dependency between the expected reward of an action and
its context. It estimates the Q value of each action using ridge regression.
It constructs a confidence region around the weights of the linear
regression model and uses this confidence ellipsoid to estimate the
uncertainty of action values.
**LinUCB-specific configs** (see also `common configs <rllib-training
.html#common-parameters>`__):
.. literalinclude:: ../../rllib/contrib/bandits/agents/lin_ucb.py
:language: python
:start-after: __sphinx_doc_begin__
:end-before: __sphinx_doc_end__
LinTS (Linear Thompson Sampling)
--------------------------------
|pytorch|
`[paper] <http://proceedings.mlr.press/v28/agrawal13.pdf>`__ `[implementation]
<https://github.com/ray-project/ray/blob/master/rllib/contrib/bandits/agents/lin_ts.py>`__
Like LinUCB, LinTS also assumes a linear dependency between the expected
reward of an action and its context and uses online ridge regression to
estimate the Q values of actions given the context. It assumes a Gaussian
prior on the weights and a Gaussian likelihood function. For deciding which
action to take, the agent samples weights for each arm, using
the posterior distributions, and plays the arm that produces the highest reward.
**LinTS-specific configs** (see also `common configs <rllib-training
.html#common-parameters>`__):
.. literalinclude:: ../../rllib/contrib/bandits/agents/lin_ts.py
:language: python
:start-after: __sphinx_doc_begin__
:end-before: __sphinx_doc_end__
.. |tensorflow| image:: tensorflow.png
:width: 24
.. |pytorch| image:: pytorch.png
:width: 24
:width: 24