update docs and UML

docs/source/actions.rst

@@ -3,6 +3,16 @@ Actions
 
 In PRL, every concept is modelized as a class. This is also true for actions which are used by the policies.
 
+.. figure:: ../figures/environment.png
+    :alt: environment
+    :align: center
+
+    The agent-environment interaction
+
+
+Design
+------
+
 
 How to use a particular action?
 ------------------------------

docs/source/approximators.rst

@@ -4,6 +4,7 @@ Approximators
 An ``Approximator`` accepts as inputs the ``State``,  ``Action``, and learning ``Model``, and connects them.
 Approximators are used by other classes in the PRL framework.
 
+
 How to use an approximator?
 ---------------------------
 

docs/source/environments.rst

@@ -1,25 +1,143 @@
 Environments
 ============
 
-- World
-- States
-- Actions
-- Rewards
+The environment is one of the main concept in imitation and reinforcement learning as defined in the following figures (inspired by [1]_):
 
-Available environments include ...
+.. figure:: ../figures/environment.png
+    :alt: environment
+    :align: center
+
+    The agent-environment interaction
 
 
-How to use an environment?
---------------------------
+The environment is notably responsible to perform a step in the world, compute the next state, compute the rewards, and others.
 
 
 Design
 ------
 
+In PRL, the environment is an abstraction layer class that regroups:
+
+- the world; an instance of ``World`` which will be used to perform a step in the world (simulator). This is called at each step performed by the environment.
+- the states: an instance of ``State`` (or a list of them). The states are updated at each time step by the environment.
+- the rewards (optional): an instance of ``Reward`` (or a list of them). It is optional because some environments like in imitation learning does not require a reward function. The reward functions are computed at each time step.
+- the terminal conditions (optional): an instance of ``TerminalCondition`` (or a list of them) that checks at each time step if the goal of the environment has been achieved. A ``TerminalCondition`` also details if the environment ended with a success or failure.
+- the initial state generators (optional): an instance of ``StateGenerator`` (or a list of them) which are called to generate the initial states each time the environment is reset.
+- the physics randomizers (optional): an instance of ``PhysicsRandomizer`` (or a list of them) to randomize the physical properties of bodies in the simulator, or the simulator itself, each time the environment is reset.
+- the actions (optional): an instance of ``Action`` (or a list of them). The actions are not used nor updated by the environment. This is left to the ``Policy`` or ``Controller``.
+
+
+By favoring `composition over inheritance <https://en.wikipedia.org/wiki/Composition_over_inheritance>`_ for the environment class, we improve the flexibility of the framework and the reuse of different modules. This leads ultimately to less code duplication, and ease the process of creating environments.
+
+
+.. figure:: ../UML/environment.png
+    :alt: UML diagram for Environment
+    :align: center
+
+    UML diagram for environment
+
+
+How to use an environment?
+--------------------------
+
+Let's assume that you have an environment where you have a manipulator, and the goal is to reach an object (such as a cube) on a table.
+
+Here is a short snippet showing the basic usage of an environment:
+
+.. code-block:: python
+    :linenos:
+
+    import pyrobolearn as prl
+
+    # define the simulator and world (and load what you want in it)
+    sim = ...
+    world = ...
+    robot = ...
+
+    # define state, action, and reward (and possibly action)
+    state = ...
+    action = ...
+    reward = ...
+
+    # you can give the reward function to your RL environment
+    # which will use it when calling `env.step()`.
+    env = prl.envs.Env(world, state, reward)
+
+    # like in OpenAI gym environments, you can reset and step in the environment
+    obs = env.reset()
+    for t in count():
+        obs, rew, done, info = env.step()
+
+
+Few notes regarding the code above:
+
+- the ``action`` can also be given to the environment but it won't be called by the environment. This is carried out by the policy(ies)/agent(s). The main reason why you can give an action to an environment is when later you will create your own environment class (that inherits from ``prl.envs.Env``), you will be able to get the states and actions for your policies in the following way:
+
+.. code-block:: python
+    :linenos:
+
+    import pyrobolearn as prl
+
+    # define your environment
+    class MyEnv(prl.envs.Env):
+        ...
+
+    # create the environment and provide possible arguments
+    env = MyEnv(args)
+
+    # get states and actions from your environment
+    states, actions = env.states, env.actions
+
+    # create policy
+    policy = Policy(states, actions)
+
+- the observation ``obs`` is a list of arrays that are returned by the environment. This is a bit different from what it is usually returned by gym environments (which is an array). The reason is that the states returned by the environment might have different dimensions (e.g. joint positions = 1D array, camera = 2D/3D array, etc) so you can not return one array.
+- You can easily update the state, reward function, world, and other modules that are given to environment. This results in less code duplication and greater flexibility.
+
+Few more examples can be found in `pyrobolearn/examples/environments <https://github.com/robotlearn/pyrobolearn/tree/master/examples/environments/>`_.
+
+If you would like other people to use your environment, implement your environment class like described in the section below.
+
 
 How to create my own environment?
 ---------------------------------
 
+Using the same example as the section above REF.
+
+.. code-block:: python
+	:linenos:
+
+	class MyEnv(Env):  # inherit from the PRL Env class
+		"""Description"""
+
+		# specify what the user is allowed to change in your environment by providing optional inputs
+		# in this case, let's say he is allowed to change the manipulator: use Franka Panda instead of Kuka
+		def __init__(self, manipulator=None, ...): 
+			# initialize the world as you would like by loading different objects in it
+			world = ...
+			...
+
+			# make sure the given manipulator is valid
+			if manipulator is None:
+				manipulator = world.load_robot('kuka_iiwa', ...)
+			if not isinstance(manipulator, prl.robots.Manipulator):
+				raise TypeError("Expecting a manipulator, instead got: {}".format(type(manipulator)))
+
+			# create the states
+			states = state1(manipulator) + ...
+
+			# create the reward
+			reward = ...
+
+			# other stuffs
+			...
+
+			# call the parent's constructor
+			super(MyEnv, self).__init__(world, states, rewards, ...)
+
+
+You normally don't have to implement anything else (like the ``step``, ``reset``, and other functions are automatically implemented based on what you provided to the parent's constructor).
+
 
 What are the differences with the OpenAI gym's environments?
 ------------------------------------------------------------
@@ -31,3 +149,9 @@ In our framework, the ``world``, ``states``, and ``rewards`` are given to the PR
 - Actions
 
 Having said that, we tried to make PRL compatible with OpenAI gym at the exception that the returned state is not a array but a list of arrays.
+
+
+References
+----------
+
+.. [1] "Reinforcement Learning: An Introduction", Sutton and Barto, 1998

docs/source/models.rst

@@ -1,12 +1,26 @@
 Models
 ======
 
-Learning models.
+Learning models versus algorithms.
 
-- DMP
+As shown on the following figures:
 
-How to use a learning model?
-----------------------------
+
+Learning models can be divided into 2 categories:
+
+- General function approximators (aka step-based learning models)
+	- Linear models
+	- Polynomial models
+	- Deep Neural Networks (DNNs)
+	- Gaussian processes (GPs)
+- Trajectory based learning models:
+	- Dynamic Movement Primitives (DMPs)
+	- Central Pattern Generators (CPGs)
+	- Gaussian Mixture Models and Gaussian Mixture Regression (GMMs/GMRs)
+	- Probabilistic Movement Primitives (ProMPs)
+	- Kernel Movement Primitives (KMPs)
+
+For few of these models, we provide a wrapper around popular libraries such as ``pytorch`` or ``gpytorch``. The other models have been reimplemented to be the most general possible.
 
 
 Design
@@ -14,8 +28,62 @@ Design
 
 Models are independent of the other elements in PRL, but are used by other elements in PRL.
 
+UML
+
+The models is notably used by approximators and policies, and their (hyper-)parameters are optimized by algorithms.
+
+
+How to use a learning model?
+----------------------------
+
+.. code-block:: python
+    :linenos:
+
+    import torch
+    import pyrobolearn as prl
+
+
+    x = torch.rand(4)
+    model = prl.models.LinearModel(num_inputs=4, num_outputs=2)
+
+    print(model.predict(y))
+
 
 How to create your own model?
 -----------------------------
 
+.. code-block:: python
+	:linenos:
 
+	import pyrobolearn as prl
+
+
+	class MyModel(prl.models.Model):
+		"""Description"""
+
+		def __init__(self, ...):
+			pass
+
+		# implement the various abstract methods
+		def ...
+
+
+Comparisons between the various models?
+---------------------------------------
+
+A question that you might have, especially if you are new to the field, what are the differences between the different models that have been proposed in the literature? In this section, I will try to provide the differences (strengths and weaknesses) of each model, and when you should favor one over another one.
+
+The below table summarizes:
+
+- General function approximator (aka step-based learning models) vs trajectory based learning models: trajectory based models accepts as inputs the time and outputs a trajectory (a sequence).
+- Parametric vs Non-parametric: In a nutshell, parametric models have parameters that are tuned by the learning algorithm based on the given dataset. Depending on the number of parameters, they might require a lot of data or to have been pretrained on similar datasets. Once trained, parametric models do not need the dataset anymore. On the other hand, non-parametric models don't have parameters but few hyper-parameters. They remember each data point in the dataset, and when given a new input they compare that new input with previous ones, and outputs an estimate based on it. Non-parametric models are very good when you don't have a lot of data points and don't have a pretrained model.
+- Linear vs Non-linear: Linear models are the most simple models that makes the least assumption about the data, but can be quite limited in their expressiveness.
+- Deterministic vs Probabilistic: Deterministic models predicts a point estimate as output without any quantity that captures the uncertainty associated with that output. Meanwhile, probabilistic models provide a probability distribution for each output. 
+- Discriminative vs Generative: discriminative models model learn the mapping ``p(y|x)`` where x is the input and y is the output, while generative models use to learn the data distribution ``p(x,y)``. Generative models are more powerful as given the prior ``p(x)`` or ``p(y)``, you can get back ``p(y|x)`` or ``p(x|y)``. Generative models might require more data.
+
+
+Future works
+------------
+
+* add methods to combine different models together
+* provide few other functionalities for the various models

docs/source/priorities.rst

@@ -0,0 +1,3 @@
+Priority Tasks
+==============
+

docs/source/pyrobolearn.rst

@@ -28,3 +28,25 @@ Hardware/Software requirements
 ------------------------------
 
 The PyRoboLearn framework has been tested on Ubuntu 16.04 and 18.04, with Python 2.7, 3.5 and 3.6.
+
+
+Design Decisions
+----------------
+
+While designing PRL, we focused on the five following features:
+
+- modularity: design a module (i.e. class) for each different concept
+- abstraction: add a layer of abstraction for combination of low-level modules
+- reusability: easy to reuse the different modules and to combine them
+- low coupling between the different modules
+- flexibility: this is mainly achieved by favoring composition over inheritance.
+
+
+The Python language has been selected.
+
+
+.. figure:: ../UML/pyrobolearn_uml.png
+	:alt: UML diagram of PyRoboLearn
+	:align: center
+
+	UML diagram of PyRoboLearn

docs/source/readme.rst

@@ -1,11 +1,6 @@
 
 .. include:: ../../README.rst
 
-Design Decisions
-================
-
-.. image:: ../UML/pyrobolearn_uml.png
-
 Citation
 ========
 

docs/source/rewards.rst

@@ -1,18 +1,121 @@
 Rewards
 =======
 
-In PRL, every concept is modelized as a class. This is also true for rewards which are returned by the environment.
+In PRL, every concept is modelized as a class. This is also true for rewards which are returned by the environment as shown in the figure below (inspired by [1]_):
+
+.. figure:: ../figures/environment.png
+    :alt: environment
+    :align: center
+
+    The agent-environment interaction
+
+
+The reward function might be defined as [1]_:
+
+- :math:`r: \mathcal{S} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}`, it returns the reward value :math:`r(s)`.
+- :math:`r: \mathcal{S} \times \mathcal{A} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}` and action :math:`a \in \mathcal{A}`, it returns the reward value :math:`r(s,a)`.
+- :math:`r: \mathcal{S} \times \mathcal{A} \times \mathcal{S} \rightarrow \mathbb{R}`: given the state :math:`s \in \mathcal{S}`, action :math:`a \in \mathcal{A}`, and next state :math:`s' \in \mathcal{S}`, it returns the reward value :math:`r(s,a,s')`.
+
+Note that the cost function is just minus the reward function, i.e. it is given by :math:`c(s,a,s') = -r(s,a,s')`.
+
+
+Design
+------
+
+In PRL, all reward functions inherit from the abstract ``Reward`` class defined in `pyrobolearn/rewards/reward.py <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/rewards>`_, and several methods and operations are provided.
+
+You can for instance:
+
+* provide the ``State`` and/or ``Action`` instances to some reward functions that will compute the reward value based on their value.
+* access to the range of the reward function.
+* add, multiply, divide, subtract, and apply basic functions such as :math:`\exp`, :math:`\cos`, :math:`\sin`, and others on reward functions. The resulting range is automatically scaled based on the operations.
+* define your own rewards/costs and reuse them in your code.
 
 
 How to use a particular reward?
 -------------------------------
 
-example of rewards
+Here is a short snippet showing the basic usage of reward functions:
+
+.. code-block:: python
+    :linenos:
+
+    import pyrobolearn as prl
+    from pyrobolearn.rewards import FixedReward, YourReward
+
+    # define the simulator and world (and load what you want in it)
+    sim = ...
+    world = ...
+    ...
+
+    # define your state / action for your reward function
+    state = ...
+    action = ...
+
+    # define the reward function
+    reward = 2 * FixedReward(3) + 0.5 * YourReward(state, action)
+
+    # print the range of the reward function
+    print(reward.range)
+
+    # compute the reward value
+    value = reward()
+    print(value)
+
+    # update the state for instance
+    state()     # this will modify the internal state data
+
+    # recompute the reward value
+    value = reward()
+    print(value)    # you will normally get a different value
+
+    # you can give the reward function to your RL environment
+    # which will use it when calling `env.step()`.
+    env = prl.envs.Env(world, state, reward)
+
+
+More examples on how to use the rewards can be found in `pyrobolearn/examples/rewards <https://github.com/robotlearn/pyrobolearn/tree/master/examples/rewards/>`_.
 
 
 How to create your own reward?
 ------------------------------
 
+In order to create your own reward, you have to inherit from ``Reward`` defined in `pyrobolearn/rewards/reward.py <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/rewards>`_.
+
+.. code-block:: python
+    :linenos:
+
+    import pyrobolearn as prl
+
+    class MyReward(prl.rewards.Reward):
+    	"""Description"""
+
+    	def __init__(self, args):
+    		# initialize your reward function based on the args
+    		...
+
+    		# gives initial value to your reward
+    		# this attribute will be used to cache the computed value
+    		self.value = 0
+
+    	def _compute(self):
+    		# compute the reward function
+    		...
+    		# save the computed value and return it
+    		self.value = ...
+    		return self.value
+
+
+Once done, you will be able to use your reward function and perform operations on it (such as addition, substraction, etc).
+
 
 FAQs
 ----
+
+* If you have any questions, please submit an issue on the `Github page <https://github.com/robotlearn/pyrobolearn>`_.
+
+
+References:
+-----------
+
+.. [1] "Reinforcement Learning: An Introduction", Sutton and Barto, 1998

docs/source/robots.rst

@@ -1,9 +1,9 @@
 Robots
 ======
 
-Robots constitute one of the main elements in the *PyRoboLearn* (PRL) framework. PRL provides a high-level abstraction and common interface to all the robots, allowing it for a better consistency and generalization between them. This allows for instance to check if one particular controller or algorithm works with other robots as well.
+Robots constitute one of the main elements in the *PyRoboLearn* (PRL) framework. PRL provides a high-level abstraction and common interface to many robots, offering better consistency and generalization between them. This allows for instance to check if one particular controller or algorithm works with other robots as well.
 
-More than 64 robots have been implemented in PRL and include various kind of robotic platforms. Among them, manipulators, biped robots, quadrupeds, hexapods, wheeled robots, quadcopters, and many others as shown below:
+More than 60+ robots are currently available in PRL covering a large range of robotic platforms. Among them, manipulators, biped robots, quadrupeds, hexapods, wheeled robots, quadcopters, and many others as shown below:
 
 
 GIF
@@ -40,7 +40,31 @@ GIF
     :width: 9%
     :alt: walkman
 
-Note that for few of them such as the ones that require the simulation of fluids such as quadcopters. The corresponding class implements the dynamical simulation. For such classes, as I did not spend too much time one it, some improvements might be needed for better realism.
+Note that for few of them such as the ones that require the simulation of fluids (e.g. quadcopters). If the simulator does not simulate fluids, the corresponding robot class implements a simple dynamics simulation. For such classes, as I did not spend too much time one it, some improvements might be needed for better realism.
+
+
+Design
+------
+
+The most abstract class is the ``Body`` class which is described in `pyrobolearn/robots/base.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/base.py>`_. From it, you can already access to multiple functionalities/attributes, such as its position and orientation. It only depends on the simulator.
+
+
+.. figure:: ../UML/robots.png
+    :alt: UML diagram for Robot
+    :align: center
+
+    UML diagram for Robot
+
+
+Inheriting from one of its child classes is the most interesting (for our purpose) ``Robot`` class, described in `robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/robot.py>`_. It is the parent class of several classes such as:
+
+- ``Manipulator`` defined in `manipulator.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/manipulator.py>`_
+- ``LeggedRobot`` defined in `legged_robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/legged_robot.py>`_
+- ``WheeledRobot`` defined in `wheeled_robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/wheeled_robot.py>`_
+- ``Hand`` defined in `hand.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/hand.py>`_
+- etc.
+
+Note that ``Robot`` only depends on the simulator interface (aggregation relationship), and is independent of other modules in PRL (at the exception of some util methods that are useful to perform some transformations).
 
 
 How to use a robot in PRL?
@@ -79,25 +103,8 @@ How to use a robot in PRL?
 
 You can check for more examples in the `examples/robots <https://github.com/robotlearn/pyrobolearn/tree/master/examples/robots>`_ folder. You can also check for `examples/kinematics <https://github.com/robotlearn/pyrobolearn/tree/master/examples/kinematics>`_ and `examples/dynamics <https://github.com/robotlearn/pyrobolearn/tree/master/examples/dynamics>`_.
 
-
-Design
-------
-
-The most abstract class is the ``Body`` class which is described in `pyrobolearn/robots/base.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/base.py>`_. From it, you can already access to multiple functionalities/attributes, such as its position and orientation. It only depends on the simulator.
-
-.. image:: ../UML/robots.png
-    :alt: UML diagram for Robot
-    :align: center
-
-Inheriting from one of its child classes is the most interesting (for our purpose) ``Robot`` class, described in `robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/robot.py>`_. It is the parent class of several classes such as:
-
-- ``Manipulator`` defined in `manipulator.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/manipulator.py>`_
-- ``LeggedRobot`` defined in `legged_robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/legged_robot.py>`_
-- ``WheeledRobot`` defined in `wheeled_robot.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/wheeled_robot.py>`_
-- ``Hand`` defined in `hand.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/hand.py>`_
-- etc.
-
-Note that ``Robot`` only depends on the simulator interface (aggregation relationship), and is independent of other modules in PRL (at the exception of some util methods that are useful to perform some transformations).
+- Kinematics
+- Dynamics
 
 
 How to create your own robot?
@@ -201,7 +208,7 @@ To illustrate how to create your own robot, let's assume you want to create a hu
             ...
 
 
-3. If you want to be able to load your robot from the world using its name (by calling ``world.load_robot('asimov')``), add the Python file ``asimov.py`` in the `pyrobolearn/robots/ <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/robots>`_ folder. The ``__init__.py`` inside that folder will automatically go through all the files and add the robots inside the ``implemented_robots`` list which is accessed by ``World``. Note that you can also accessed to this list by calling ``pyrobolearn.robots.implemented_robots``. If you also want to be able to call your robot using ``from pyrobolearn.robots import Asimov``, you will have to add the line ``from .asimov import Asimov`` in the `pyrobolearn/robots/__init__.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/__init__.py>`_.
+3. If you want to be able to load your robot from the world using its name (by calling ``world.load_robot('asimov')``), add the Python file ``asimov.py`` in the `pyrobolearn/robots/ <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/robots>`_ folder. The ``__init__.py`` inside that folder will automatically go through all the files and add the robots inside the ``implemented_robots`` list which is accessed by ``World``. Note that you can also access this list by calling ``pyrobolearn.robots.implemented_robots``. If you also want to be able to call your robot using ``from pyrobolearn.robots import Asimov``, you will have to add the line ``from .asimov import Asimov`` in the `pyrobolearn/robots/__init__.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/robots/__init__.py>`_.
 
 4. Now, you can call your robot in the framework.
 
@@ -237,13 +244,15 @@ FAQs and Troubleshootings
 -------------------------
 
 - The mass/inertia matrix of some links are not correct in the simulator, what should I do?
-    * If you use the Bullet simulator (which uses ``pybullet``), you have to specify the mass and inertia matrix for each link. If a link doesn't have these attributes defined, pybullet automatically attribute a mass of 1kg and an identity inertia matrix (which is ridiculous huge). Normally, links without a mass and inertia matrices defined in a URDF file are dummy links that are used to represent a reference frame. To set a reasonable inertia matrix, please refer to `"Adding Physical and Collision Properties to a URDF Model" <http://wiki.ros.org/urdf/Tutorials/Adding%20Physical%20and%20Collision%20Properties%20to%20a%20URDF%20Model>`_ and `"Inertial parameters of triangle meshes" <http://gazebosim.org/tutorials?tut=inertia&cat=build_robot>`_.
-    * It is possible that some masses / inertia matrices have not been correctly set in the original URDF. I cleaned most of the URDF files but some links might have escaped my attention. Please open an issue on `Github <https://github.com/robotlearn/pyrobolearn>`_, or check the 2 `links <http://wiki.ros.org/urdf/Tutorials/Adding%20Physical%20and%20Collision%20Properties%20to%20a%20URDF%20Model>`_ `above <http://gazebosim.org/tutorials?tut=inertia&cat=build_robot>`_ on how to set reasonable inertia values.
+    - If you use the Bullet simulator (which uses ``pybullet``), you have to specify the mass and inertia matrix for each link. If a link doesn't have these attributes defined, pybullet automatically attribute a mass of 1kg and an identity inertia matrix (which is ridiculous huge). Normally, links without a mass and inertia matrices defined in a URDF file are dummy links that are used to represent a reference frame. To set a reasonable inertia matrix, please refer to `"Adding Physical and Collision Properties to a URDF Model" <http://wiki.ros.org/urdf/Tutorials/Adding%20Physical%20and%20Collision%20Properties%20to%20a%20URDF%20Model>`_ and `"Inertial parameters of triangle meshes" <http://gazebosim.org/tutorials?tut=inertia&cat=build_robot>`_.
+    - It is possible that some masses / inertia matrices have not been correctly set in the original URDF. I cleaned most of the URDF files but some links might have escaped my attention. Please open an issue on `Github <https://github.com/robotlearn/pyrobolearn>`_, or check the 2 `links <http://wiki.ros.org/urdf/Tutorials/Adding%20Physical%20and%20Collision%20Properties%20to%20a%20URDF%20Model>`_ `above <http://gazebosim.org/tutorials?tut=inertia&cat=build_robot>`_ on how to set reasonable inertia values.
 
 - How to convert a xacro file to a URDF file? Type ``rosrun xacro xacro --inorder path/to/<robot>.urdf.xacro > <robot>.urdf`` or ``rosrun xacro xacro.py --inorder path/to/<robot>.urdf.xacro > <robot>.urdf``
 
 - When I set the ``fixed_base`` to ``False``, the robot has still a fixed base, what is happening? The first link (often called base_link or world_link in most URDF files) shouldn't have a mass/inertia of zero, this causes the robot to have a fixed base. Remove the corresponding tag from the urdf.
 
+- What are the differences when a robot has a floating-based and a fixed base? When the robot has a floating base, the total number of degrees of freedom becomes 6 + the number of actuated joints. This appears when computing the Jacobian and Inertia matrices.
+
 - I noticed that some functionalities are missing in one of the robot class? I probably forgot to implement it. Please open an issue on `Github <https://github.com/robotlearn/pyrobolearn>`_ or create a pull request.
 
 - There is an error in one of the functionalities? Or, I have another question or want to suggest an improvement? Please open an issue on `Github <https://github.com/robotlearn/pyrobolearn>`_ or a create a pull request.

docs/source/simulators.rst

@@ -1,13 +1,31 @@
 Simulators
 ==========
 
-The simulator is the starting point in the *%PyRoboLearn** (PRL) framework. To avoid a tight coupling with a particular simulator, a ``Simulator`` interface class (from which all the other simulators inherit from) has been implemented. Other .
+The simulator is the starting point in the **PyRoboLearn** (PRL) framework. To avoid a tight coupling with a particular simulator, a ``Simulator`` interface class (from which all the other simulators inherit from) has been implemented. Other .
 
-We provide the ``Bullet`` interface.
+We provide the ``Bullet`` interface, which uses the ``pybullet`` library.
 
 The general idea is that you would be able to change the simulator if you wish without having to modify any other lines of code. See example below.
 
 
+Design
+------
+
+The important goal when designing the simulators was that it should be a stand alone interface.
+
+
+.. figure:: ../UML/simulator.png
+    :alt: UML diagram for Simulators
+    :align: center
+
+    UML diagram for simulators
+
+
+Currently, the only fully functional simulator is ``Bullet``. While other simulators have been considered such as Gazebo (with ROS), Mujoco, Dart, and others, few of them required a particular license or do not have strong Python bindings.
+
+Note that each simulator implements static methods which provide information about the simulator itself; if it can for instance simulate fluids or not, etc.
+
+
 How to use a particular simulator in PRL?
 -----------------------------------------
 
@@ -39,21 +57,21 @@ For the moment, the only fully operational interface is ``Bullet``. Some few oth
 You can check for more examples in the `examples/simulators <https://github.com/robotlearn/pyrobolearn/tree/master/examples/simulators>`_ folder.
 
 
-Design
-------
-
-The important goal when designing the simulators was that it should be a stand alone interface.
-
-UML
-
-
-
 How to create an interface to a simulator?
 ------------------------------------------
 
 To create your own Simulator, you have to inherit from the ``Simulator`` class defined in `pyrobolearn/simulators/simulator.py <https://github.com/robotlearn/pyrobolearn/blob/master/pyrobolearn/simulators/simulator.py>`_.
 
 
+.. code-block:: python
+
+    import pyrobolearn as prl
+
+
+    class MySimulator(prl.simulators.Simulator):
+        """Description"""
+        # implement all the abstract methods in Simulator
+
 
 FAQs and Troubleshootings
 -------------------------
@@ -69,6 +87,7 @@ Future works
 ------------
 
 My main objectives for future works are the implementation of:
+
 - the Mujoco interface; I originally did not start with it as it is closed-source and requires a License. However, it is used a lot in research and thus it could be interesting to have it as well.
 - the Gazebo-ROS interface; a part has already been implemented but it is far from over.
 - the Isaac interface if Nvidia provided a nice Python API.

docs/source/states.rst

@@ -3,16 +3,106 @@ States
 
 In PRL, every concept is modelized as a class. This is also true for states which are returned by the environment.
 
+.. figure:: ../figures/environment.png
+    :alt: environment
+    :align: center
+
+    The agent-environment interaction
+
+
+States are given to the policy and the environment. The environment is responsible to update them while policies read their ``data`` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the ``states()`` which updates the ``data`` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the ``data`` of the various ``states`` without having to call the ``states()`` itself to update their values.
+
+States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
+
+
+Design
+------
+
+UML
+
 
 How to use a particular state?
 ------------------------------
 
-example of states with robot
+Let's assume you have a quadruped robot, and you would like to get its joint positions, velocities, and base position.
+
+.. code-block:: python
+    :linenos:
+
+    from itertools import count
+    import pyrobolearn as prl
+   	from pyrobolearn.states import BasePositionState, JointPositionState, JointVelocityState
+
+
+    # create simulator
+    sim = prl.simulators.Bullet()
+
+    # load robot
+    robot = prl.robots.HyQ2Max(sim)
+
+    # create the states
+    base_pos_state = BasePositionState(robot)
+    joint_pos_state = JointPositionState(robot, joint_ids=robot.legs)  # you can specify which joints you would like to get the position
+    joint_vel_state = JointVelocityState(robot, joint_ids=robot.legs)  # you can specify which joints you would like to get the velocity
+
+    state = base_pos_state + joint_pos_state + joint_vel_state
+
+    # run simulation
+    for t in count():
+    	# call and print the state
+    	print(state())
+
+    	# perform a step in the world
+    	world.step(sim.dt)
+
+
+All the states accept also as inputs:
+
+- ``window_size``: size (by default, it is set to one)
+- ``ticks``: the number of simulation ticks to sleep before getting the next state. 
+
+In the example above, for joint states, you could also specify the joints that you would like to get the states from by setting ``joint_ids``. Note that in order to be able to generalize to other robots, avoid to give manually the joint ids but instead gives an attribute of the robot, like, ``robot.legs`` (wich is a list containing the joint ids associated with each leg). 
+
+Now, let's assume that you forgot to include the robot's base orientation and its linear/angular velocity in the state. In other frameworks, it is very likely that you would have to change manually the state and everything that depends on it (the policy / value function approximator which accepts as input the state, the step function in the environment which compute the next state, possibly the reward function which is often based on the current state, etc). In PRL, everything is automatized, and thus setting:
+
+.. code-block:: python
+
+	state = state + BaseOrientationState(robot) + BaseLinearVelocity(robot)
+
+will automatically results the other components to update their input size or because this new state is provided to them.
 
 
 How to create your own state?
 -----------------------------
 
+Let's assume that you want to create a state that accepts as inputs the game controller .
+
+.. code-block:: python
+    :linenos:
+
+    import pyrobolearn as prl
+
+
+    class MyState(prl.states.State):  # inherit from the abstract State class
+    	"""Description"""
+
+    	def __init__(self, ...):
+
+    		super(MyState, self).__init__(...)
+
 
 FAQs
 ----
+
+
+Other functionalities
+---------------------
+
+- `State generator <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/states/generators>`_: generate a state (used as initial state generator)
+- `State processor <https://github.com/robotlearn/pyrobolearn/tree/master/pyrobolearn/states/processors>`_: process the given state (before giving it to another model such as a policy)
+
+
+Future works
+------------
+
+* add a ``rate`` attribute to the states which is used when we set the real-time on the simulator. Or better, using the ``ticks`` and ``sim.dt`` infer the rate.

docs/source/worlds.rst

@@ -4,6 +4,22 @@ Worlds
 The world is the second important item in PRL; it is, with the ``Body`` class (see next section), the only class that can access the simulator. As it name implies, it allows you to create a world in the simulator, load various objects in it, and change the world's and objects' physical properties. From it, you can also access to the main camera (if the GUI is enabled in the simulator), and move it as you wish. The world can be seen as a wrapper around the simulator which provides you extra functionalities where each function calls different methods of the simulator. Finally, the world also allows you to load and generate terrains.
 
 
+Design
+------
+
+As it can be seen on the UML diagram below the ``World`` depends on the ``Simulator`` and the various ``Body`` (see next section) loaded in it (as well as few util functions).
+
+
+.. figure:: ../UML/world.png
+    :alt: UML diagram for World
+    :align: center
+
+    UML diagram for world
+
+
+Later, we will see that world is notably given to the environment along with the states and rewards.
+
+
 How to use the world in PRL?
 ----------------------------
 
@@ -39,18 +55,6 @@ Note that you can get access to the world camera, and change its position and or
 For more examples, you can check the `examples/worlds <https://github.com/robotlearn/pyrobolearn/tree/master/examples/worlds>`_ folder.
 
 
-Design
-------
-
-As it can be seen on the UML diagram below the ``World`` depends on the ``Simulator`` and the various ``Body`` (see next section) loaded in it (as well as few util functions).
-
-
-UML picture
-
-
-Later, we will see that world is notably given to the environment along with the states and rewards.
-
-
 How to create your own world?
 -----------------------------
 
@@ -75,3 +79,4 @@ Where can I find 3d models to load in the world?
 - `Gazebo database <https://bitbucket.org/osrf/gazebo_models/src/default/>`_
 - `Turbosquid <www.turbosquid.com>`_
 - `free3d <free3d.com>`_
+- `sketchfab <sketchfab.com>`_