diff --git a/examples/states/README.md b/examples/states/README.md new file mode 100644 index 0000000..a537157 --- /dev/null +++ b/examples/states/README.md @@ -0,0 +1,52 @@ +## States + +In this folder, you will find some examples on how to use the `pyrobolearn.states.*` states. + +States are classes defined outside the environment and are basically containers. They can be updated by calling them `state()` (same as `state.read()`), or by setting their `data` variable. + +They can be combined using the addition operator. For instance, `s_a = s1 + s2`, `s_b = s2 + s3`, and `s=s_a + s_b`. Calling `s_a()` will update the states `s1` and `s2`, and thus the data contained in `s_b` as well (as it contains a pointer to `s2` which has been updated). You can just call `s()` to update in one loop `s1, s2, s3` altogether (`s_a` and `s_b` will reflect that change because they contain a pointer to these states `s1, s2, s3`). + +States are given to the policy and the environment. The environment is responsible to update them while policies read their `data` and feed it to the underlying learning model. In the case we use a physics simulator like PyBullet, the environment performs one step in the simulation and calls the `states()` which updates the `data` they contained. Instead, if you have a dynamical model function, the environment can call this one to update the `data` of the various `states` without having to call the `states()` itself to update their values. + +States can also be given to dynamical models (which predicts the next state given the current state and last action), value function approximators (which predicts a scalar value given a state and possibly an action), reward functions, etc. + + +### Simple Example + +```python +import pyrobolearn.states as states + +s1 = states.CumulativeTimeState() +s2 = states.AbsoluteTimeState() +s = s1 + s2 +print(s) + +# update s1 +s1.read() # or s1() +print(s1) +print(s) # just s1 changed + +# update s1 and s2 by calling s +s() +print(s) +print(s1) +print(s2) + +# get the data +print(s.data) # this will return a list of 2 arrays; each one of shape (1,). The size of the list is equal to the number of states that it contains +print(s1.data) # this will return a list with one array of shape (1,) +print(s.merged_data) # this will return a merged state; it will merge the states that have the same dimensions together and return a list of arrays which has a size equal to the number of different dimensions. The arrays inside that list are ordered by their dimensionality in an ascending way. +``` + +## What to test first? + +Try to launch `basics.py` first. + + +## For the programmer + +Why states are defined outside and not inside the environments like usually done in `gym.envs`. States are defined outside for a better modularity, reusability, flexibility, and lower coupling. +* Why better modularity? Because you define a module for each possible state which you can combine at your taste later on. +* Why better reusability? Because it avoids you to define how to read similar state in different environments which often lead to code duplication. +* Why lower coupling? Coupling between two modules measures how much they are dependent on each other. There is a lower coupling, because instead of having a composition relationship between the modules we have an aggregation relationship (see [UML Association vs Aggregation vs Composition](https://www.visual-paradigm.com/guide/uml-unified-modeling-language/uml-aggregation-vs-composition/)). That is, because states are defined outside of the environment and given to the environment, even if we destroyed the environment, the states still exist. +* Why better flexibility? Because we favor [composition over inheritance](https://en.wikipedia.org/wiki/Composition_over_inheritance). you can combine different states as you wish, give different states to different policies, and provide them at the end to the environment (which will update them). diff --git a/examples/states/basics.py b/examples/states/basics.py new file mode 100644 index 0000000..06bf085 --- /dev/null +++ b/examples/states/basics.py @@ -0,0 +1,44 @@ +#!/usr/bin/env python +"""Test different states +""" + +import pyrobolearn.states as states + +s1 = states.CumulativeTimeState() +s2 = states.AbsoluteTimeState() +s = s1 + s2 + +print("\ns = s1 + s2 = {}".format(s)) + +# update s1 +print("\nUpdating state s1 by calling s1():") +s1.read() # or s1() +print("s after calling s1() = {}".format(s)) +print("s1 after calling s1() = {}".format(s1)) +print("s2 after calling s1() = {}".format(s2)) + +# update s1 and s2 by calling s +print("\nUpdating the combined state s=(s1+s2) by calling s():") +s() +print("s after calling s() = {}".format(s)) +print("s1 after calling s() = {}".format(s1)) +print("s2 after calling s() = {}".format(s2)) + +# get the data +print("\nGetting the state data:") + +# this will return a list of 2 arrays; each one of shape (1,). The size of the list +# is equal to the number of states that it contains +print("s.data = {}".format(s.data)) + +# this will return a list with one array of shape (1,). +print("s1.data = {}".format(s1.data)) + +# this will return a merged state; it will merge the states that have the same dimensions +# together and return a list of arrays which has a size equal to the number of different +# dimensions. The arrays inside that list are ordered by their dimensionality in an +# ascending way. +print("s.merged_data = {}".format(s.merged_data)) + +# this will return the same as s1.data because it just have one state +print("s1.merged_data = {}\n".format(s1.merged_data))