States
In this folder, you will find some examples on how to use the pyrobolearn.states.* states.
States are classes defined outside the environment and are basically containers. They can be updated by calling them state() (same as state.read()), or by setting their data variable.
They can be combined using the addition operator. For instance, s_a = s1 + s2, s_b = s2 + s3, and s=s_a + s_b. Calling s_a() will update the states s1 and s2, and thus the data contained in s_b as well (as it contains a pointer to s2 which has been updated). You can just call s() to update in one loop s1, s2, s3 altogether (s_a and s_b will reflect that change because they contain a pointer to these states s1, s2, s3).
States are given to the policy and the environment. The environment is responsible to update them while policies read their data and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the states() which updates the data they contained. Instead, if you have a dynamical model function, the environment can call this one to update the data of the various states without having to call the states() itself to update their values.
States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc.
Simple Example
import pyrobolearn.states as states
s1 = states.CumulativeTimeState()
s2 = states.AbsoluteTimeState()
s = s1 + s2
print(s)
# update s1
s1.read() # or s1()
print(s1)
print(s) # just s1 changed
# update s1 and s2 by calling s
s()
print(s)
print(s1)
print(s2)
# get the data
print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains
print(s1.data) # this will return a list with one array of shape (1,)
print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way.
What to test first?
Try to launch basics.py first.
For the programmer
Why states are defined outside and not inside the environments like usually done in gym.envs. States are defined outside for a better modularity, reusability, flexibility, and lower coupling.
- Why better modularity? Because you define a module for each possible state which you can combine at your taste later on.
- Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication.
- Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see UML Association vs Aggregation vs Composition). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist.
- Why better flexibility? Because we favor composition over inheritance. you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them).