mirror of
https://github.com/wassname/pyrobolearn.git
synced 2026-08-25 11:23:51 +08:00
122 lines
4.1 KiB
ReStructuredText
122 lines
4.1 KiB
ReStructuredText
Rewards
|
|
=======
|
|
|
|
In PRL, every concept is modelized as a class. This is also true for rewards which are returned by the environment as shown in the figure below (inspired by [1]_):
|
|
|
|
.. figure:: ../figures/environment.png
|
|
:alt: environment
|
|
:align: center
|
|
|
|
The agent-environment interaction
|
|
|
|
|
|
The reward function might be defined as [1]_:
|
|
|
|
- :math:`r: \mathcal{S} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}`, it returns the reward value :math:`r(s)`.
|
|
- :math:`r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}` and action :math:`a \in \mathcal{A}`, it returns the reward value :math:`r(s,a)`.
|
|
- :math:`r: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}`, action :math:`a \in \mathcal{A}`, and next state :math:`s' \in \mathcal{S}`, it returns the reward value :math:`r(s,a,s')`.
|
|
|
|
Note that the cost function is just minus the reward function, i.e. it is given by :math:`c(s,a,s') = -r(s,a,s')`.
|
|
|
|
|
|
Design
|
|
------
|
|
|
|
In PRL, all reward functions inherit from the abstract ``Reward`` class defined in `pyrobolearn/rewards/reward.py <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/rewards>`_, and several methods and operations are provided.
|
|
|
|
You can for instance:
|
|
|
|
* provide the ``State`` and/or ``Action`` instances to some reward functions that will compute the reward value based on their value.
|
|
* access to the range of the reward function.
|
|
* add, multiply, divide, subtract, and apply basic functions such as :math:`\exp`, :math:`\cos`, :math:`\sin`, and others on reward functions. The resulting range is automatically scaled based on the operations.
|
|
* define your own rewards/costs and reuse them in your code.
|
|
|
|
|
|
How to use a particular reward?
|
|
-------------------------------
|
|
|
|
Here is a short snippet showing the basic usage of reward functions:
|
|
|
|
.. code-block:: python
|
|
:linenos:
|
|
|
|
import pyrobolearn as prl
|
|
from pyrobolearn.rewards import FixedReward, YourReward
|
|
|
|
# define the simulator and world (and load what you want in it)
|
|
sim = ...
|
|
world = ...
|
|
...
|
|
|
|
# define your state / action for your reward function
|
|
state = ...
|
|
action = ...
|
|
|
|
# define the reward function
|
|
reward = 2 * FixedReward(3) + 0.5 * YourReward(state, action)
|
|
|
|
# print the range of the reward function
|
|
print(reward.range)
|
|
|
|
# compute the reward value
|
|
value = reward()
|
|
print(value)
|
|
|
|
# update the state for instance
|
|
state() # this will modify the internal state data
|
|
|
|
# recompute the reward value
|
|
value = reward()
|
|
print(value) # you will normally get a different value
|
|
|
|
# you can give the reward function to your RL environment
|
|
# which will use it when calling `env.step()`.
|
|
env = prl.envs.Env(world, state, reward)
|
|
|
|
|
|
More examples on how to use the rewards can be found in `pyrobolearn/examples/rewards <https://github.com/robotlearn/pyrobolearn/tree/master/examples/rewards/>`_.
|
|
|
|
|
|
How to create your own reward?
|
|
------------------------------
|
|
|
|
In order to create your own reward, you have to inherit from ``Reward`` defined in `pyrobolearn/rewards/reward.py <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/rewards>`_.
|
|
|
|
.. code-block:: python
|
|
:linenos:
|
|
|
|
import pyrobolearn as prl
|
|
|
|
class MyReward(prl.rewards.Reward):
|
|
"""Description"""
|
|
|
|
def __init__(self, args):
|
|
# initialize your reward function based on the args
|
|
...
|
|
|
|
# gives initial value to your reward
|
|
# this attribute will be used to cache the computed value
|
|
self.value = 0
|
|
|
|
def _compute(self):
|
|
# compute the reward function
|
|
...
|
|
# save the computed value and return it
|
|
self.value = ...
|
|
return self.value
|
|
|
|
|
|
Once done, you will be able to use your reward function and perform operations on it (such as addition, substraction, etc).
|
|
|
|
|
|
FAQs
|
|
----
|
|
|
|
* If you have any questions, please submit an issue on the `Github page <https://github.com/robotlearn/pyrobolearn>`_.
|
|
|
|
|
|
References:
|
|
-----------
|
|
|
|
.. [1] "Reinforcement Learning: An Introduction", Sutton and Barto, 1998
|