fix and update few minor functionalities

This commit is contained in:
Brian Delhaisse
2019-06-29 02:58:03 +02:00
parent b9cf231ee9
commit 55e4d8323e
17 changed files with 253 additions and 112 deletions
+92
View File
@@ -0,0 +1,92 @@
Environments
============
In this folder, we provide examples on how to define and use environments which are notably useful for imitation and reinforcement learning.
An environment is defined as the following figures (inspired by [1]_):
.. image:: ../../docs/figures/environment.png
:alt: environment
:align: center
In PRL, the environment is an abstraction layer class that regroups:
- the world; an instance of ``World`` which will be used to perform a step in the world (simulator). This is called at each step performed by the environment.
- the states: an instance of ``State`` (or a list of them). The states are updated at each time step by the environment.
- the rewards (optional): an instance of ``Reward`` (or a list of them). It is optional because some environments like in imitation learning does not require a reward function. The reward functions are computed at each time step.
- the terminal conditions (optional): an instance of ``TerminalCondition`` (or a list of them) that checks at each time step if the goal of the environment has been achieved. A ``TerminalCondition`` also details if the environment ended with a success or failure.
- the initial state generators (optional): an instance of ``StateGenerator`` (or a list of them) which are called to generate the initial states each time the environment is reset.
- the physics randomizers (optional): an instance of ``PhysicsRandomizer`` (or a list of them) to randomize the physical properties of bodies in the simulator, or the simulator itself, each time the environment is reset.
- the actions (optional): an instance of ``Action`` (or a list of them). The actions are not used nor updated by the environment. This is left to the ``Policy`` or ``Controller``.
By favoring `composition over inheritance <https://en.wikipedia.org/wiki/Composition_over_inheritance>`_ for the environment class, we improve the flexibility of the framework and the reuse of different modules.
This leads ultimately to less code duplication, and ease the process of creating environments.
Here is a short snippet showing the basic usage of an environment:
.. code-block:: python
:linenos:
import pyrobolearn as prl
# define the simulator and world (and load what you want in it)
sim = ...
world = ...
robot = ...
# define state, action, and reward (and possibly action)
state = ...
action = ...
reward = ...
# you can give the reward function to your RL environment
# which will use it when calling `env.step()`.
env = prl.envs.Env(world, state, reward)
# like in OpenAI gym environments, you can reset and step in the environment
obs = env.reset()
for t in count():
obs, rew, done, info = env.step()
Few notes regarding the code above:
- the ``action`` can also be given to the environment but it won't be called by the environment. This is carried out by the policy(ies)/agent(s). The main reason why you can give an action to an environment is when later you will create your own environment class (that inherits from ``prl.envs.Env``), you will be able to get the states and actions for your policies in the following way:
.. code-block:: python
:linenos:
import pyrobolearn as prl
# define your environment
class MyEnv(prl.envs.Env):
...
# create the environment and provide possible arguments
env = MyEnv(args)
# get states and actions from your environment
states, actions = env.states, env.actions
# create policy
policy = Policy(states, actions)
- the observation ``obs`` is a list of arrays that are returned by the environment. This is a bit different from what it is usually returned by gym environments (which is an array). The reason is that the states returned by the environment might have different dimensions (e.g. joint positions = 1D array, camera = 2D/3D array, etc) so you can not return one array.
- You can easily update the state, reward function, world, and other modules that are given to environment. This results in less code duplication and greater flexibility.
For more info, please check the documentation.
Examples
~~~~~~~~
Here are few examples that you can find in this folder that better demonstrate how to use the environment:
1. ``basics.py``: show the flexibility of how to build an environment and use it.
2. ``manipulator.py``: show how to define an environment where the goal is to reach a target object using a manipulator.
References:
.. [1] "Reinforcement Learning: An Introduction", Sutton and Barto, 1998
@@ -58,7 +58,7 @@ sim = prl.simulators.Bullet()
# create basic world (with a floor and gravity enabled by default)
world = prl.worlds.BasicWorld(sim)
# load quadcopter
# load wheeled robot
robot = prl.robots.Epuck(sim, position=[0., 0.])
world.load_robot(robot)
+1 -1
View File
@@ -58,7 +58,7 @@ sim = prl.simulators.Bullet()
# create basic world (with a floor and gravity enabled by default)
world = prl.worlds.BasicWorld(sim)
# load quadcopter
# load wheeled robot
robot = prl.robots.Epuck(sim, position=[0., 0.])
world.load_robot(robot)
-52
View File
@@ -1,52 +0,0 @@
## States
In this folder, you will find some examples on how to use the `pyrobolearn.states.*` states.
States are classes defined outside the environment and are basically containers. They can be updated by calling them `state()` (same as `state.read()`), or by setting their `data` variable.
They can be combined using the addition operator. For instance, `s_a = s1 + s2`, `s_b = s2 + s3`, and `s=s_a + s_b`. Calling `s_a()` will update the states `s1` and `s2`, and thus the data contained in `s_b` as well (as it contains a pointer to `s2` which has been updated). You can just call `s()` to update in one loop `s1, s2, s3` altogether (`s_a` and `s_b` will reflect that change because they contain a pointer to these states `s1, s2, s3`).
States are given to the policy and the environment. The environment is responsible to update them while policies read their `data` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the `states()` which updates the `data` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the `data` of the various `states` without having to call the `states()` itself to update their values.
States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
### Simple Example
```python
import pyrobolearn.states as states
s1 = states.CumulativeTimeState()
s2 = states.AbsoluteTimeState()
s = s1 + s2
print(s)
# update s1
s1.read() # or s1()
print(s1)
print(s) # just s1 changed
# update s1 and s2 by calling s
s()
print(s)
print(s1)
print(s2)
# get the data
print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains
print(s1.data) # this will return a list with one array of shape (1,)
print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way.
```
## What to test first?
Try to launch `basics.py` first.
## For the programmer
Why states are defined outside and not inside the environments like usually done in `gym.envs`. States are defined outside for a better modularity, reusability, flexibility, and lower coupling.
* Why better modularity? Because you define a module for each possible state which you can combine at your taste later on.
* Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication.
* Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see [UML Association vs Aggregation vs Composition](https://www.visual-paradigm.com/guide/uml-unified-modeling-language/uml-aggregation-vs-composition/)). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist.
* Why better flexibility? Because we favor [composition over inheritance](https://en.wikipedia.org/wiki/Composition_over_inheritance). you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them).
+62
View File
@@ -0,0 +1,62 @@
States
======
In this folder, you will find some examples on how to use the ``pyrobolearn.states.*`` states.
States are classes defined outside the environment and are basically containers. They can be updated by calling them ``state()`` (same as ``state.read()``), or by setting their ``data`` variable.
They can be combined using the addition operator. For instance, ``s_a = s1 + s2``, ``s_b = s2 + s3``, and ``s=s_a + s_b``. Calling ``s_a()`` will update the states ``s1`` and ``s2``, and thus the data contained in ``s_b`` as well (as it contains a pointer to ``s2`` which has been updated). You can just call ``s()`` to update in one loop ``s1, s2, s3`` altogether (``s_a`` and ``s_b`` will reflect that change because they contain a pointer to these states ``s1, s2, s3``).
States are given to the policy and the environment. The environment is responsible to update them while policies read their ``data`` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the ``states()`` which updates the ``data`` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the ``data`` of the various ``states`` without having to call the ``states()`` itself to update their values.
States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
Here are few examples that you can find in this folder:
1. ``basics.py``: demonstrate the various features of the ``State`` class.
2. ``robot.py``: get the joint states of a specific robot and print them.
3. ``world.py``: get the pose state of an object loaded in the world.
4. ``sensor.py``: get the state of a sensor.
5. ``interface.py``: get the state from a game controller interface.
Simple Example
--------------
.. code-block:: python
:linenos:
import pyrobolearn.states as states
s1 = states.CumulativeTimeState()
s2 = states.AbsoluteTimeState()
s = s1 + s2
print(s)
# update s1
s1.read() # or s1()
print(s1)
print(s) # just s1 changed
# update s1 and s2 by calling s
s()
print(s)
print(s1)
print(s2)
# get the data
print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains
print(s1.data) # this will return a list with one array of shape (1,)
print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way.
For the programmer
------------------
Why states are defined outside and not inside the environments like usually done in ``gym.envs``. States are defined outside for a better modularity, reusability, flexibility, and lower coupling.
- Why better modularity? Because you define a module for each possible state which you can combine at your taste later on.
- Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication.
- Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see `UML Association vs Aggregation vs Composition <https://www.visual-paradigm.com/guide/uml-unified-modeling-language/uml-aggregation-vs-composition/>`_). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist.
- Why better flexibility? Because we favor `composition over inheritance <https://en.wikipedia.org/wiki/Composition_over_inheritance>`_. you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them).
+6 -6
View File
@@ -3,25 +3,25 @@
from .optimizer import Optimizer
# import scipy optimizer
# from scipy_optimizer import Scipy
# from .scipy_optimizer import Scipy
# import nlopt optimizer
# from nlopt_optimizer import NLopt
# from .nlopt_optimizer import NLopt
# import ipopt optimizer
# from ipopt_optimizer import IPopt
# from .ipopt_optimizer import IPopt
# import QP solvers
# from qpsolvers_optimizer import QP
from .qpsolvers_optimizer import QP
# import CMA-ES optimizer
# from cma_optimizer import CMAES
# import Bayesian Optimizer
# from gpyopt_optimizer import BayesianOptimizer
# from .gpyopt_optimizer import BayesianOptimizer
# import torch optimizers
from .torch_optimizer import *
# import Contact-Invariant Optimizer
# from cio import CIO
# from .cio import CIO
+3 -3
View File
@@ -57,9 +57,9 @@ class NLopt(Optimizer):
Initialize the non-linear optimizer.
Args:
method:
submethod:
seed:
method (str): primary optimization method to be used.
submethod (str): sub-optimization method to be used in the primary optimization method.
seed (None, int): random seed
*args:
**kwargs:
"""
+30 -3
View File
@@ -111,7 +111,8 @@ class QP(Optimizer):
Initialize the QP solver.
Args:
method (str): ['cvxopt', 'cvxpy', 'ecos', 'gurobi', 'mosek', 'osqp', 'qpoases', 'quadprog']
method (str): QP method/library to use. Select between ['cvxopt', 'cvxpy', 'ecos', 'gurobi', 'mosek',
'osqp', 'qpoases', 'quadprog']
"""
super(QP, self).__init__(*args, **kwargs)
@@ -126,16 +127,26 @@ class QP(Optimizer):
methods = ['cvxopt', 'osqp', 'quadprog']
self.sym_proj = True if self.method in set(methods) else False
def is_symmetric(self, X, tol=1e-8):
##################
# Static Methods #
##################
@staticmethod
def is_symmetric(X, tol=1e-8):
"""Check if the given matrix is symmetric."""
return np.allclose(X, X.T, atol=tol)
###########
# Methods #
###########
def optimize(self, P, q, x0=None, G=None, h=None, A=None, b=None):
r"""
Optimize the given quadratic problem.
.. math::
\min_{x \in R^n} \frac{1}{2} x^T P x + q^T x
\min_{x \in \mathbb{R}^N} \frac{1}{2} x^T P x + q^T x
subject to
@@ -144,6 +155,22 @@ class QP(Optimizer):
Gx \leq h
Ax = b
Args:
P (np.array[N,N]): matrix used in the QP objective function where `N` is the size of the vector `x` being
optimized.
q (np.array[N]): vector used in the QP objective function where `N` is the size of the vector `x` being
optimized.
G (np.array[M,N]): matrix used in the inequality constraint, where `M` is the number of inequalities, and
`N` is the size of the vector `x` being optimized. Note that if you have lower and upper bounds for
the vector `x`, you can set :attr:`G` to be the concatenation of :math:`[-I, I]^\top`, where :math:`I`
is the identity matrix.
h (np.array[M]): vector used in the inequality constraint, where `M` is the number of inequalities. Note
that if you have lower and upper bounds for the vector `x`, you can set :attr:`h` to be the
concatenation of :math:`[-b_l^\top, b_u^\top]`, where :math:`b_l` and :math:`b_u` are the lower and
upper bounds respectively.
A (np.array[K,N]): matrix used in the equality constraint.
b (np.array[K,N]): vector used in the equality constraint.
Returns:
np.array: QP solution
"""
+2 -1
View File
@@ -1,6 +1,7 @@
# import abstract reward class and operations
from .reward import *
from .reward import Reward, ceil, cos, cosh, degrees, exp, expm1, floor, frexp, isinf, isnan, log, log10, log1p, \
radians, sin, sinh, sqrt, tan, tanh, trunc
# import basic rewards
from .basic_rewards import *
+5 -3
View File
@@ -42,13 +42,15 @@ class FixedReward(Reward):
if not isinstance(value, (int, float)):
raise TypeError("Expecting a number")
self.value = value
self.range = (value, value) if range is None else range
self.range = (value, value) if range is None or (isinstance(range, (tuple, list)) and len(range) == 0) \
else range
if value < self.range[0] or value > self.range[1]:
raise ValueError("The given value (={}) is not in the specified range = {}".format(value, self.range))
def __repr__(self):
return '%s(%s)' % (self.__class__.__name__, str(self.value))
def __str__(self):
"""Return a string describing the reward."""
return '%s(%s, range=%s)' % (self.__class__.__name__, str(self.value), str(self.range))
def _compute(self):
return self.value
+1 -1
View File
@@ -426,7 +426,7 @@ class DistanceCost(Cost):
cost to change.
"""
def __init__(self):
def __init__(self, body1, body2):
super(DistanceCost, self).__init__()
def loss(self, object1, object2):
+25 -23
View File
@@ -15,12 +15,14 @@ Dependencies:
- `pyrobolearn.actions`
"""
import numpy as np
import collections
import operator
import copy
# from pyrobolearn.rewards.objective import Objective
from pyrobolearn.states import *
from pyrobolearn.actions import *
from pyrobolearn.states import State
from pyrobolearn.actions import Action
__author__ = "Brian Delhaisse"
@@ -291,27 +293,27 @@ class Reward(object):
"""Compute the reward function."""
return self.compute() # **kwargs)
def __copy__(self):
"""Return a shallow copy of the reward function. This can be overridden in the child class."""
return self.__class__(state=self.state, action=self.action, rewards=self.rewards, range=self.range)
def __deepcopy__(self, memo={}):
"""Return a deep copy of the reward function. This can be overridden in the child class.
Args:
memo (dict): memo dictionary of objects already copied during the current copying pass
"""
if self in memo:
return memo[self]
state = copy.deepcopy(self.state, memo)
action = copy.deepcopy(self.action, memo)
rewards = [copy.deepcopy(reward, memo) for reward in self.rewards]
range = copy.deepcopy(self.range)
reward = self.__class__(state=state, action=action, rewards=rewards, range=range)
memo[self] = reward
return reward
# def __copy__(self):
# """Return a shallow copy of the reward function. This can be overridden in the child class."""
# return self.__class__(state=self.state, action=self.action, rewards=self.rewards, range=self.range)
#
# def __deepcopy__(self, memo={}):
# """Return a deep copy of the reward function. This can be overridden in the child class.
#
# Args:
# memo (dict): memo dictionary of objects already copied during the current copying pass
# """
# if self in memo:
# return memo[self]
#
# state = copy.deepcopy(self.state, memo)
# action = copy.deepcopy(self.action, memo)
# rewards = [copy.deepcopy(reward, memo) for reward in self.rewards]
# range = copy.deepcopy(self.range)
# reward = self.__class__(state=state, action=action, rewards=rewards, range=range)
#
# memo[self] = reward
# return reward
# for unary and binary operators, see `__init__()` method.
+17 -17
View File
@@ -154,9 +154,9 @@ class LeggedRobot(Robot):
.. math::
x_{CoP} = \frac{\sum_i x_i f^i_n}{\sum{i} f^i_n}
y_{CoP} = \frac{\sum_i y_i f^i_n}{\sum{i} f^i_n}
z_{CoP} = \frac{\sum_i z_i f^i_n}{\sum{i} f^i_n}
x_{CoP} = \frac{\sum_i x_i f^i_n}{\sum_i f^i_n}
y_{CoP} = \frac{\sum_i y_i f^i_n}{\sum_i f^i_n}
z_{CoP} = \frac{\sum_i z_i f^i_n}{\sum_i f^i_n}
where :math:`[x_i, y_i, z_i]` are the coordinates of the contact point :math:`i` on which the normal force
:math:`f^i_n` acts.
@@ -172,8 +172,8 @@ class LeggedRobot(Robot):
np.array[3], None: center of pressure. None if the robot is not in contact with the ground.
References:
[1] "Postural Stability of Biped Robots and Foot-Rotation Index (FRI) Point", Goswami, 1999
[1] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
- [1] "Postural Stability of Biped Robots and Foot-Rotation Index (FRI) Point", Goswami, 1999
- [2] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
Implications", Popovic et al., 2005
"""
if floor_id is not None:
@@ -254,10 +254,10 @@ class LeggedRobot(Robot):
np.array[3], None: zero-moment point. None if the ground reaction force in z is 0.
References:
[1] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
- [1] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
Implications", Popovic et al., 2005
[2] "Biped Walking Pattern Generation by using Preview Control of ZMP", Kajita et al., 2003
[3] "Exploiting Angular Momentum to Enhance Bipedal Center-of-Mass Control", Hofmann et al., 2009
- [2] "Biped Walking Pattern Generation by using Preview Control of ZMP", Kajita et al., 2003
- [3] "Exploiting Angular Momentum to Enhance Bipedal Center-of-Mass Control", Hofmann et al., 2009
"""
# if we need to update the CoM
if update_com:
@@ -327,8 +327,8 @@ class LeggedRobot(Robot):
- the FRI coincides with the ZMP when the foot is stationary. [1]
References:
[1] "Postural Stability of Biped Robots and the Foot-Rotation Indicator (FRI) Point", Goswami, 1999
[2] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
- [1] "Postural Stability of Biped Robots and the Foot-Rotation Indicator (FRI) Point", Goswami, 1999
- [2] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
Implications", Popovic et al., 2005
"""
raise NotImplementedError
@@ -362,7 +362,7 @@ class LeggedRobot(Robot):
np.array[3], None: centroidal moment pivot point. None if the ground reaction force in z is 0.
References:
[1] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
- [1] "Ground Reference Points in Legged Locomotion: Definitions, Biological Trajectories and Control
Implications", Popovic et al., 2005
"""
# update the CoM
@@ -410,7 +410,7 @@ class LeggedRobot(Robot):
# \dot{y}, \dot{z}]` are the CoM position and velocity, :math:`b > 0` is a time-constant of the DCM dynamics.
#
# References:
# [1] "Three-dimensional Bipedal Walking Control Based on Divergent Component of Motion", Englsberger et
# - [1] "Three-dimensional Bipedal Walking Control Based on Divergent Component of Motion", Englsberger et
# al., 2015
# """
# pass
@@ -458,7 +458,7 @@ class LeggedRobot(Robot):
lifetime (float): lifetime of the support polygon before it disappears.
References:
[1] "A Universal Stability Criterion of the Foot Contact of Legged Robots- Adios ZMP"
- [1] "A Universal Stability Criterion of the Foot Contact of Legged Robots- Adios ZMP"
"""
# get contact points between the robot's links and the floor
points = self.sim.get_contact_points(body1=self.id, body2=floor_id)
@@ -501,8 +501,8 @@ class LeggedRobot(Robot):
height (float): maximum height of the cone in the simulator.
References:
[1] https://scaron.info/teaching/friction-cones.html
[2] "Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone
- [1] https://scaron.info/teaching/friction-cones.html
- [2] "Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone
for Rectangular Support Areas", Caron et al., 2015
"""
filename = os.path.dirname(__file__) + '/../worlds/meshes/cone.obj'
@@ -564,8 +564,8 @@ class LeggedRobot(Robot):
height (float): maximum height of the pyramid in the simulator.
References:
[1] https://scaron.info/teaching/friction-cones.html
[2] "Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone
- [1] https://scaron.info/teaching/friction-cones.html
- [2] "Stability of Surface Contacts for Humanoid Robots: Closed-Form Formulae of the Contact Wrench Cone
for Rectangular Support Areas", Caron et al., 2015
"""
filename = os.path.dirname(__file__) + '/../worlds/meshes/pyramid.obj'
@@ -2,3 +2,6 @@
# import
from .bridge_mousekeyboard_world import BridgeMouseKeyboardWorld
from .bridge_mousekeyboard_imitation_task import BridgeMouseKeyboardImitationTask
from .bridge_mousekeyboard_wheeled import BridgeMouseKeyboardWheeledRobot, \
BridgeMouseKeyboardDifferentialWheeledRobot, BridgeMouseKeyboardAckermannWheeledRobot
from .bridge_mousekeyboard_quadcopter import BridgeMouseKeyboardQuadcopter
@@ -191,7 +191,7 @@ class BridgeMouseKeyboardDifferentialWheeledRobot(BridgeMouseKeyboardWheeledRobo
Initialize the Bridge between a Mouse-Keyboard interface and a differential wheeled robot instance.
Args:
robot (AckermannWheeledRobot): wheeled robot instance.
robot (DifferentialWheeledRobot): wheeled robot instance.
interface (None, MouseKeyboardInterface): mouse keyboard interface. If None, it will create one.
camera (WorldCamera): world camera instance. This will allow the user to switch between first-person and
third-person view. If None, it will create an instance of it.
@@ -0,0 +1,4 @@
# TELEGRAM:
# - https://telegram.org/
# - https://github.com/telegramdesktop/tdesktop
# - https://github.com/python-telegram-bot/python-telegram-bot