mirror of
https://github.com/wassname/pyrobolearn.git
synced 2026-09-17 12:40:23 +08:00
fix and update few minor functionalities
This commit is contained in:
Brian Delhaisse
@@ -0,0 +1,92 @@
|
||||
Environments
|
||||
============
|
||||
|
||||
In this folder, we provide examples on how to define and use environments which are notably useful for imitation and reinforcement learning.
|
||||
|
||||
An environment is defined as the following figures (inspired by [1]_):
|
||||
|
||||
.. image:: ../../docs/figures/environment.png
|
||||
:alt: environment
|
||||
:align: center
|
||||
|
||||
In PRL, the environment is an abstraction layer class that regroups:
|
||||
|
||||
- the world; an instance of ``World`` which will be used to perform a step in the world (simulator). This is called at each step performed by the environment.
|
||||
- the states: an instance of ``State`` (or a list of them). The states are updated at each time step by the environment.
|
||||
- the rewards (optional): an instance of ``Reward`` (or a list of them). It is optional because some environments like in imitation learning does not require a reward function. The reward functions are computed at each time step.
|
||||
- the terminal conditions (optional): an instance of ``TerminalCondition`` (or a list of them) that checks at each time step if the goal of the environment has been achieved. A ``TerminalCondition`` also details if the environment ended with a success or failure.
|
||||
- the initial state generators (optional): an instance of ``StateGenerator`` (or a list of them) which are called to generate the initial states each time the environment is reset.
|
||||
- the physics randomizers (optional): an instance of ``PhysicsRandomizer`` (or a list of them) to randomize the physical properties of bodies in the simulator, or the simulator itself, each time the environment is reset.
|
||||
- the actions (optional): an instance of ``Action`` (or a list of them). The actions are not used nor updated by the environment. This is left to the ``Policy`` or ``Controller``.
|
||||
|
||||
|
||||
By favoring `composition over inheritance <https://en.wikipedia.org/wiki/Composition_over_inheritance>`_ for the environment class, we improve the flexibility of the framework and the reuse of different modules.
|
||||
This leads ultimately to less code duplication, and ease the process of creating environments.
|
||||
|
||||
Here is a short snippet showing the basic usage of an environment:
|
||||
|
||||
.. code-block:: python
|
||||
:linenos:
|
||||
|
||||
import pyrobolearn as prl
|
||||
|
||||
# define the simulator and world (and load what you want in it)
|
||||
sim = ...
|
||||
world = ...
|
||||
robot = ...
|
||||
|
||||
# define state, action, and reward (and possibly action)
|
||||
state = ...
|
||||
action = ...
|
||||
reward = ...
|
||||
|
||||
# you can give the reward function to your RL environment
|
||||
# which will use it when calling `env.step()`.
|
||||
env = prl.envs.Env(world, state, reward)
|
||||
|
||||
# like in OpenAI gym environments, you can reset and step in the environment
|
||||
obs = env.reset()
|
||||
for t in count():
|
||||
obs, rew, done, info = env.step()
|
||||
|
||||
|
||||
Few notes regarding the code above:
|
||||
|
||||
- the ``action`` can also be given to the environment but it won't be called by the environment. This is carried out by the policy(ies)/agent(s). The main reason why you can give an action to an environment is when later you will create your own environment class (that inherits from ``prl.envs.Env``), you will be able to get the states and actions for your policies in the following way:
|
||||
|
||||
.. code-block:: python
|
||||
:linenos:
|
||||
|
||||
import pyrobolearn as prl
|
||||
|
||||
# define your environment
|
||||
class MyEnv(prl.envs.Env):
|
||||
...
|
||||
|
||||
# create the environment and provide possible arguments
|
||||
env = MyEnv(args)
|
||||
|
||||
# get states and actions from your environment
|
||||
states, actions = env.states, env.actions
|
||||
|
||||
# create policy
|
||||
policy = Policy(states, actions)
|
||||
|
||||
- the observation ``obs`` is a list of arrays that are returned by the environment. This is a bit different from what it is usually returned by gym environments (which is an array). The reason is that the states returned by the environment might have different dimensions (e.g. joint positions = 1D array, camera = 2D/3D array, etc) so you can not return one array.
|
||||
- You can easily update the state, reward function, world, and other modules that are given to environment. This results in less code duplication and greater flexibility.
|
||||
|
||||
|
||||
For more info, please check the documentation.
|
||||
|
||||
|
||||
Examples
|
||||
~~~~~~~~
|
||||
|
||||
Here are few examples that you can find in this folder that better demonstrate how to use the environment:
|
||||
|
||||
1. ``basics.py``: show the flexibility of how to build an environment and use it.
|
||||
2. ``manipulator.py``: show how to define an environment where the goal is to reach a target object using a manipulator.
|
||||
|
||||
References:
|
||||
|
||||
.. [1] "Reinforcement Learning: An Introduction", Sutton and Barto, 1998
|
||||
@@ -58,7 +58,7 @@ sim = prl.simulators.Bullet()
|
||||
# create basic world (with a floor and gravity enabled by default)
|
||||
world = prl.worlds.BasicWorld(sim)
|
||||
|
||||
# load quadcopter
|
||||
# load wheeled robot
|
||||
robot = prl.robots.Epuck(sim, position=[0., 0.])
|
||||
world.load_robot(robot)
|
||||
|
||||
|
||||
@@ -58,7 +58,7 @@ sim = prl.simulators.Bullet()
|
||||
# create basic world (with a floor and gravity enabled by default)
|
||||
world = prl.worlds.BasicWorld(sim)
|
||||
|
||||
# load quadcopter
|
||||
# load wheeled robot
|
||||
robot = prl.robots.Epuck(sim, position=[0., 0.])
|
||||
world.load_robot(robot)
|
||||
|
||||
|
||||
@@ -1,52 +0,0 @@
|
||||
## States
|
||||
|
||||
In this folder, you will find some examples on how to use the `pyrobolearn.states.*` states.
|
||||
|
||||
States are classes defined outside the environment and are basically containers. They can be updated by calling them `state()` (same as `state.read()`), or by setting their `data` variable.
|
||||
|
||||
They can be combined using the addition operator. For instance, `s_a = s1 + s2`, `s_b = s2 + s3`, and `s=s_a + s_b`. Calling `s_a()` will update the states `s1` and `s2`, and thus the data contained in `s_b` as well (as it contains a pointer to `s2` which has been updated). You can just call `s()` to update in one loop `s1, s2, s3` altogether (`s_a` and `s_b` will reflect that change because they contain a pointer to these states `s1, s2, s3`).
|
||||
|
||||
States are given to the policy and the environment. The environment is responsible to update them while policies read their `data` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the `states()` which updates the `data` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the `data` of the various `states` without having to call the `states()` itself to update their values.
|
||||
|
||||
States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
|
||||
|
||||
|
||||
### Simple Example
|
||||
|
||||
```python
|
||||
import pyrobolearn.states as states
|
||||
|
||||
s1 = states.CumulativeTimeState()
|
||||
s2 = states.AbsoluteTimeState()
|
||||
s = s1 + s2
|
||||
print(s)
|
||||
|
||||
# update s1
|
||||
s1.read() # or s1()
|
||||
print(s1)
|
||||
print(s) # just s1 changed
|
||||
|
||||
# update s1 and s2 by calling s
|
||||
s()
|
||||
print(s)
|
||||
print(s1)
|
||||
print(s2)
|
||||
|
||||
# get the data
|
||||
print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains
|
||||
print(s1.data) # this will return a list with one array of shape (1,)
|
||||
print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way.
|
||||
```
|
||||
|
||||
## What to test first?
|
||||
|
||||
Try to launch `basics.py` first.
|
||||
|
||||
|
||||
## For the programmer
|
||||
|
||||
Why states are defined outside and not inside the environments like usually done in `gym.envs`. States are defined outside for a better modularity, reusability, flexibility, and lower coupling.
|
||||
* Why better modularity? Because you define a module for each possible state which you can combine at your taste later on.
|
||||
* Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication.
|
||||
* Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see [UML Association vs Aggregation vs Composition](https://www.visual-paradigm.com/guide/uml-unified-modeling-language/uml-aggregation-vs-composition/)). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist.
|
||||
* Why better flexibility? Because we favor [composition over inheritance](https://en.wikipedia.org/wiki/Composition_over_inheritance). you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them).
|
||||
@@ -0,0 +1,62 @@
|
||||
States
|
||||
======
|
||||
|
||||
In this folder, you will find some examples on how to use the ``pyrobolearn.states.*`` states.
|
||||
|
||||
States are classes defined outside the environment and are basically containers. They can be updated by calling them ``state()`` (same as ``state.read()``), or by setting their ``data`` variable.
|
||||
|
||||
They can be combined using the addition operator. For instance, ``s_a = s1 + s2``, ``s_b = s2 + s3``, and ``s=s_a + s_b``. Calling ``s_a()`` will update the states ``s1`` and ``s2``, and thus the data contained in ``s_b`` as well (as it contains a pointer to ``s2`` which has been updated). You can just call ``s()`` to update in one loop ``s1, s2, s3`` altogether (``s_a`` and ``s_b`` will reflect that change because they contain a pointer to these states ``s1, s2, s3``).
|
||||
|
||||
States are given to the policy and the environment. The environment is responsible to update them while policies read their ``data`` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the ``states()`` which updates the ``data`` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the ``data`` of the various ``states`` without having to call the ``states()`` itself to update their values.
|
||||
|
||||
States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
|
||||
|
||||
|
||||
Here are few examples that you can find in this folder:
|
||||
|
||||
1. ``basics.py``: demonstrate the various features of the ``State`` class.
|
||||
2. ``robot.py``: get the joint states of a specific robot and print them.
|
||||
3. ``world.py``: get the pose state of an object loaded in the world.
|
||||
4. ``sensor.py``: get the state of a sensor.
|
||||
5. ``interface.py``: get the state from a game controller interface.
|
||||
|
||||
|
||||
Simple Example
|
||||
--------------
|
||||
|
||||
.. code-block:: python
|
||||
:linenos:
|
||||
|
||||
import pyrobolearn.states as states
|
||||
|
||||
s1 = states.CumulativeTimeState()
|
||||
s2 = states.AbsoluteTimeState()
|
||||
s = s1 + s2
|
||||
print(s)
|
||||
|
||||
# update s1
|
||||
s1.read() # or s1()
|
||||
print(s1)
|
||||
print(s) # just s1 changed
|
||||
|
||||
# update s1 and s2 by calling s
|
||||
s()
|
||||
print(s)
|
||||
print(s1)
|
||||
print(s2)
|
||||
|
||||
# get the data
|
||||
print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains
|
||||
print(s1.data) # this will return a list with one array of shape (1,)
|
||||
print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way.
|
||||
|
||||
|
||||
For the programmer
|
||||
------------------
|
||||
|
||||
Why states are defined outside and not inside the environments like usually done in ``gym.envs``. States are defined outside for a better modularity, reusability, flexibility, and lower coupling.
|
||||
|
||||
- Why better modularity? Because you define a module for each possible state which you can combine at your taste later on.
|
||||
- Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication.
|
||||
- Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see `UML Association vs Aggregation vs Composition <https://www.visual-paradigm.com/guide/uml-unified-modeling-language/uml-aggregation-vs-composition/>`_). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist.
|
||||
- Why better flexibility? Because we favor `composition over inheritance <https://en.wikipedia.org/wiki/Composition_over_inheritance>`_. you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them).
|
||||
Reference in New Issue
Block a user