mirror of
https://github.com/wassname/pyrobolearn.git
synced 2026-09-09 11:31:38 +08:00
update state readme + add interface states
This commit is contained in:
Brian Delhaisse
@@ -0,0 +1,21 @@
|
||||
## Reward Processors
|
||||
|
||||
This folder contains code to process the rewards that are returned by the environment. These wraps the original reward, and when called returned a processed reward value. Reward processors are also considered as rewards (as they inherit from the `Reward` class). Wrapping the rewards allows to use different processors for different rewards, and inheriting from `Reward` allows to use the various operations defined in that class. You can, for instance, add two reward processors.
|
||||
|
||||
Here is a pseudo-code to illustrate the above points:
|
||||
```python
|
||||
r1 = Reward1(args)
|
||||
r2 = Reward2(args)
|
||||
|
||||
reward = 0.5 * r1 + r2
|
||||
proc_reward_1 = RewardProcessor(reward, args)
|
||||
proc_reward_2 = 0.5 * RewardProcessor1(r1, args1) + RewardProcessor2(r2, args2)
|
||||
|
||||
# to compute the reward, just call the reward which would normally provide different values
|
||||
reward_value = reward()
|
||||
proc_reward_1_value = proc_reward_1()
|
||||
proc_reward_2_value = proc_reward_2()
|
||||
```
|
||||
|
||||
One of the important reward processors used in the RL field (as described in other libraries such as the `baselines.common.vec_env.vec_normalize.py`) is the `GammaStandardizeRewardProcessor` which computes the return :math:`R = r + \gamma * R ` at each time step, then computes a running standard deviation on that return, and finally divides the reward by that standard deviation before returning it.
|
||||
|
||||
@@ -23,3 +23,6 @@ from .gym_states import *
|
||||
#
|
||||
# # import state processors
|
||||
# from .processors import *
|
||||
#
|
||||
# # import interface states
|
||||
# from .interfaces import *
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
## State generators
|
||||
|
||||
This folder contains code to generate states. The initial state generator generates the initial state which is returned by the environment when calling `env.reset()`. Note that the state can be generated in a deterministic manner or randomly based on a distribution.
|
||||
|
||||
@@ -0,0 +1,4 @@
|
||||
## Interface states
|
||||
|
||||
This folder contains the states that get the information from interfaces defined in `pyrobolearn.tools.interfaces`; these include joystick / game controllers, cameras (such as webcams, kinect, etc), and others. To demonstrate its usefulness, let's consider a state that reads the joystick position, and let's assume we want to train a policy that receives as input that state, and outputs an action to move the wheels of a robot. In the training mode, the joystick state can generate deterministically or randomly (based on a probability distribution such as :math:`s_t \sim p(.)`, or :math:`s_{t+1} = s_t + \epsilon` with :math:`\epsilon \sim p(.)`), a joystick position that will be given to the policy. Note that the state is also provided to the reward function which can account on how to compute the reward value based on, for instance, the joystick position state and the robot base position state. In the test phase, the joystick state returns the position read from the joystick interface.
|
||||
|
||||
@@ -0,0 +1,12 @@
|
||||
|
||||
# import interface state
|
||||
from .interface_state import InterfaceState
|
||||
|
||||
# import mouse keyboard states
|
||||
from .mouse_keyboard_state import *
|
||||
|
||||
# import joystick states
|
||||
from .joystick_states import *
|
||||
|
||||
# import webcam states
|
||||
from .webcam_state import *
|
||||
@@ -0,0 +1,79 @@
|
||||
#!/usr/bin/env python
|
||||
"""Define the abstract interface state class.
|
||||
|
||||
The interface state allows to get the state information about input interfaces defined in
|
||||
`pyrobolearn.tools.interfaces`.
|
||||
|
||||
Dependencies:
|
||||
- `pyrobolearn.states`
|
||||
- `pyrobolearn.robots`
|
||||
- `pyrobolearn.tools.interfaces.InputInterface`
|
||||
"""
|
||||
|
||||
from pyrobolearn.states import State
|
||||
from pyrobolearn.tools.interfaces.interface import InputInterface, InputOutputInterface
|
||||
|
||||
|
||||
__author__ = "Brian Delhaisse"
|
||||
__copyright__ = "Copyright 2018, PyRoboLearn"
|
||||
__credits__ = ["Brian Delhaisse"]
|
||||
__license__ = "MIT"
|
||||
__version__ = "1.0.0"
|
||||
__maintainer__ = "Brian Delhaisse"
|
||||
__email__ = "briandelhaisse@gmail.com"
|
||||
__status__ = "Development"
|
||||
|
||||
|
||||
class InterfaceState(State):
|
||||
r"""Input Interface state.
|
||||
|
||||
This is a state that reads the data from an input interface (such as a mouse, keyboard, microphone, webcam /
|
||||
kinect, game controllers, VR/AR controllers, etc).
|
||||
"""
|
||||
|
||||
def __init__(self, interface):
|
||||
"""
|
||||
Initialize the (input) interface state.
|
||||
|
||||
Args:
|
||||
interface (InputInterface): input interface.
|
||||
"""
|
||||
super(InterfaceState, self).__init__()
|
||||
|
||||
# set the interface
|
||||
if not isinstance(interface, (InputInterface, InputOutputInterface)):
|
||||
raise TypeError("Expecting the given 'interface' to be an instance of `InputInterface` or "
|
||||
"`InputOutputInterface`, instead got: {}".format(type(interface)))
|
||||
self._interface = interface
|
||||
|
||||
# set training mode
|
||||
self._training_mode = False
|
||||
|
||||
##############
|
||||
# Properties #
|
||||
##############
|
||||
|
||||
@property
|
||||
def interface(self):
|
||||
"""Return the input interface instance."""
|
||||
return self._interface
|
||||
|
||||
@property
|
||||
def interface_in_thread(self):
|
||||
"""Return True if the interface is running in another thread."""
|
||||
return self._interface.use_thread
|
||||
|
||||
###########
|
||||
# Methods #
|
||||
###########
|
||||
|
||||
def _reset(self):
|
||||
"""Reset the state and return the data."""
|
||||
pass
|
||||
|
||||
def _read(self):
|
||||
"""Read the next state or generate a state based on the :attr:`training_mode` that the state is in.
|
||||
If :attr:`interface.use_thread` is False and :attr:`training_mode` is True, then it has to update the
|
||||
interface as well.
|
||||
"""
|
||||
pass
|
||||
@@ -0,0 +1,170 @@
|
||||
#!/usr/bin/env python
|
||||
"""Define the joystick interface states.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
||||
from pyrobolearn.states.interfaces.interface_state import InterfaceState
|
||||
import pyrobolearn.tools.interfaces.controllers as controllers
|
||||
|
||||
|
||||
__author__ = "Brian Delhaisse"
|
||||
__copyright__ = "Copyright 2018, PyRoboLearn"
|
||||
__credits__ = ["Brian Delhaisse"]
|
||||
__license__ = "MIT"
|
||||
__version__ = "1.0.0"
|
||||
__maintainer__ = "Brian Delhaisse"
|
||||
__email__ = "briandelhaisse@gmail.com"
|
||||
__status__ = "Development"
|
||||
|
||||
|
||||
class JoystickState(InterfaceState):
|
||||
r"""Joystick Interface state.
|
||||
|
||||
This is a state that reads the data from a joystick or game controller interface.
|
||||
"""
|
||||
|
||||
def __init__(self, interface):
|
||||
"""
|
||||
Initialize the game controller (input) interface state.
|
||||
|
||||
Args:
|
||||
interface (GameControllerInterface): game controller input interface.
|
||||
"""
|
||||
if not isinstance(interface, controllers.GameControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `GameControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(InterfaceState, self).__init__(interface)
|
||||
|
||||
def _reset(self):
|
||||
"""Reset the state and return the data."""
|
||||
pass
|
||||
|
||||
def _read(self):
|
||||
"""Read the next state or generate a state based on the :attr:`training_mode` that the state is in.
|
||||
If :attr:`interface.use_thread` is False and :attr:`training_mode` is True, then it has to update the
|
||||
interface as well.
|
||||
"""
|
||||
pass
|
||||
|
||||
|
||||
class XboxControllerState(JoystickState):
|
||||
r"""Xbox Controller state.
|
||||
"""
|
||||
|
||||
def __init__(self, interface):
|
||||
"""
|
||||
Initialize the Xbox controller state.
|
||||
|
||||
Args:
|
||||
interface (XboxControllerInterface): Xbox controller input interface.
|
||||
"""
|
||||
if not isinstance(interface, controllers.xbox.XboxControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `XboxControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(XboxControllerState, self).__init__(interface)
|
||||
|
||||
|
||||
class Xbox360ControllerState(XboxControllerState):
|
||||
r"""Xbox 360 controller state.
|
||||
"""
|
||||
|
||||
def __init__(self, interface=None, use_thread=False):
|
||||
"""
|
||||
Initialize the Xbox 360 controller state.
|
||||
|
||||
Args:
|
||||
interface (Xbox360ControllerInterface, None): Xbox 360 controller input interface. If None, it will
|
||||
initialize the interface.
|
||||
use_thread (bool): If True, it will run the interface in a thread. Otherwise, it will run the interface
|
||||
everytime it is called.
|
||||
"""
|
||||
if interface is None:
|
||||
interface = controllers.xbox.Xbox360ControllerInterface(use_thread=use_thread)
|
||||
if not isinstance(interface, controllers.xbox.Xbox360ControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `Xbox360ControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(Xbox360ControllerState, self).__init__(interface)
|
||||
|
||||
|
||||
class XboxOneControllerState(XboxControllerState):
|
||||
r"""Xbox 360 controller state.
|
||||
"""
|
||||
|
||||
def __init__(self, interface=None, use_thread=False):
|
||||
"""
|
||||
Initialize the Xbox One controller state.
|
||||
|
||||
Args:
|
||||
interface (XboxOneControllerInterface, None): Xbox 360 controller input interface. If None, it will
|
||||
initialize the interface.
|
||||
use_thread (bool): If True, it will run the interface in a thread. Otherwise, it will run the interface
|
||||
everytime it is called.
|
||||
"""
|
||||
if interface is None:
|
||||
interface = controllers.xbox.XboxOneControllerInterface(use_thread=use_thread)
|
||||
if not isinstance(interface, controllers.xbox.XboxOneControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `XboxOneControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(XboxOneControllerState, self).__init__(interface)
|
||||
|
||||
|
||||
class PSControllerState(JoystickState):
|
||||
r"""Playstation Controller State.
|
||||
"""
|
||||
|
||||
def __init__(self, interface):
|
||||
"""
|
||||
Initialize the PS controller state.
|
||||
|
||||
Args:
|
||||
interface (PSControllerInterface): Playstation controller input interface.
|
||||
"""
|
||||
if not isinstance(interface, controllers.playstation.PSControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `PSControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(PSControllerState, self).__init__(interface)
|
||||
|
||||
|
||||
class PS3ControllerState(PSControllerState):
|
||||
r"""PlayStation 3 controller state.
|
||||
"""
|
||||
|
||||
def __init__(self, interface=None, use_thread=False):
|
||||
"""
|
||||
Initialize the PlayStation 3 controller state.
|
||||
|
||||
Args:
|
||||
interface (PS3ControllerInterface, None): PlayStation 3 controller input interface. If None, it will
|
||||
initialize the interface.
|
||||
use_thread (bool): If True, it will run the interface in a thread. Otherwise, it will run the interface
|
||||
everytime it is called.
|
||||
"""
|
||||
if interface is None:
|
||||
interface = controllers.playstation.PS3ControllerInterface(use_thread=use_thread)
|
||||
if not isinstance(interface, controllers.playstation.PS3ControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `PS3ControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(PS3ControllerState, self).__init__(interface)
|
||||
|
||||
|
||||
class PS4ControllerState(PSControllerState):
|
||||
r"""PlayStation 4 controller state.
|
||||
"""
|
||||
|
||||
def __init__(self, interface=None, use_thread=False):
|
||||
"""
|
||||
Initialize the PlayStation 4 controller state.
|
||||
|
||||
Args:
|
||||
interface (PS4ControllerInterface, None): PlayStation 4 controller input interface. If None, it will
|
||||
initialize the interface.
|
||||
use_thread (bool): If True, it will run the interface in a thread. Otherwise, it will run the interface
|
||||
everytime it is called.
|
||||
"""
|
||||
if interface is None:
|
||||
interface = controllers.playstation.PS4ControllerInterface(use_thread=use_thread)
|
||||
if not isinstance(interface, controllers.playstation.PS4ControllerInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `PS3ControllerInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(PS4ControllerState, self).__init__(interface)
|
||||
@@ -0,0 +1,48 @@
|
||||
#!/usr/bin/env python
|
||||
"""Define the mouse keyboard interface states.
|
||||
"""
|
||||
|
||||
import numpy as np
|
||||
|
||||
from pyrobolearn.states.interfaces.interface_state import InterfaceState
|
||||
import pyrobolearn.tools.interfaces.mouse_keyboard as mouse_keyboard
|
||||
|
||||
|
||||
__author__ = "Brian Delhaisse"
|
||||
__copyright__ = "Copyright 2018, PyRoboLearn"
|
||||
__credits__ = ["Brian Delhaisse"]
|
||||
__license__ = "MIT"
|
||||
__version__ = "1.0.0"
|
||||
__maintainer__ = "Brian Delhaisse"
|
||||
__email__ = "briandelhaisse@gmail.com"
|
||||
__status__ = "Development"
|
||||
|
||||
|
||||
class MouseKeyboardState(InterfaceState):
|
||||
r"""Mouse Keyboard Interface state.
|
||||
|
||||
This is a state that reads the data from a mouse keyboard controller interface.
|
||||
"""
|
||||
|
||||
def __init__(self, interface):
|
||||
"""
|
||||
Initialize the mouse keyboard (input) interface state.
|
||||
|
||||
Args:
|
||||
interface (MouseKeyboardInterface): mouse keyboard input interface.
|
||||
"""
|
||||
if not isinstance(interface, mouse_keyboard.MouseKeyboardInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `MouseKeyboardInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(InterfaceState, self).__init__(interface)
|
||||
|
||||
def _reset(self):
|
||||
"""Reset the state and return the data."""
|
||||
pass
|
||||
|
||||
def _read(self):
|
||||
"""Read the next state or generate a state based on the :attr:`training_mode` that the state is in.
|
||||
If :attr:`interface.use_thread` is False and :attr:`training_mode` is True, then it has to update the
|
||||
interface as well.
|
||||
"""
|
||||
pass
|
||||
@@ -0,0 +1,94 @@
|
||||
#!/usr/bin/env python
|
||||
"""Define the webcam interface states.
|
||||
"""
|
||||
|
||||
# TODO: debug
|
||||
|
||||
import numpy as np
|
||||
|
||||
from pyrobolearn.states.interfaces.interface_state import InterfaceState
|
||||
import pyrobolearn.tools.interfaces.camera.webcam as webcam
|
||||
|
||||
|
||||
__author__ = "Brian Delhaisse"
|
||||
__copyright__ = "Copyright 2018, PyRoboLearn"
|
||||
__credits__ = ["Brian Delhaisse"]
|
||||
__license__ = "MIT"
|
||||
__version__ = "1.0.0"
|
||||
__maintainer__ = "Brian Delhaisse"
|
||||
__email__ = "briandelhaisse@gmail.com"
|
||||
__status__ = "Development"
|
||||
|
||||
|
||||
class WebcamState(InterfaceState):
|
||||
r"""Webcam Interface state.
|
||||
|
||||
This is a state that reads the data from a webcam or game controller interface.
|
||||
"""
|
||||
|
||||
def __init__(self, interface=None, use_thread=True, sleep_dt=1./10, verbose=False):
|
||||
"""
|
||||
Initialize the webcam (input) interface state.
|
||||
|
||||
Args:
|
||||
interface (WebcamInterface, None): webcam input interface. If None, it will initialize the interface.
|
||||
use_thread (bool): If True, it will run the interface in a thread. Otherwise, it will run the interface
|
||||
everytime it is called.
|
||||
sleep_dt (float): If :attr:`use_thread` is True, it will sleep the specified amount before acquiring the
|
||||
next sample.
|
||||
verbose (bool): If True, it will print information about the state of the webcam interface.
|
||||
"""
|
||||
if interface is None:
|
||||
interface = webcam.WebcamInterface(use_thread=use_thread, sleep_dt=sleep_dt, verbose=verbose)
|
||||
if not isinstance(interface, webcam.WebcamInterface):
|
||||
raise TypeError("Expecting the given interface to be an instance of `WebcamInterface`, instead "
|
||||
"got: {}".format(type(interface)))
|
||||
super(WebcamState, self).__init__(interface)
|
||||
|
||||
def _reset(self):
|
||||
"""Reset the state and return the data."""
|
||||
self.data = self.interface.frame
|
||||
|
||||
def _read(self):
|
||||
"""Read the next state or generate a state based on the :attr:`training_mode` that the state is in.
|
||||
If :attr:`interface.use_thread` is False and :attr:`training_mode` is True, then it has to update the
|
||||
interface as well.
|
||||
"""
|
||||
# if self.in_training_mode and not self.interface_in_thread
|
||||
self.data = self.interface.frame
|
||||
|
||||
|
||||
# Tests
|
||||
if __name__ == '__main__':
|
||||
from itertools import count
|
||||
import matplotlib.pyplot as plt
|
||||
|
||||
# create webcam state
|
||||
state = WebcamState()
|
||||
|
||||
# plotting using matplotlib in interactive mode
|
||||
fig = plt.figure()
|
||||
plot = None
|
||||
plt.ion() # interactive mode on
|
||||
|
||||
for _ in count():
|
||||
# # if don't use thread call `step` or `run` (note that `run` returns the frame but not
|
||||
# interface.step()
|
||||
|
||||
# get the frame and plot it with matplotlib
|
||||
state()
|
||||
frame = state.data
|
||||
if len(frame) > 0:
|
||||
frame = frame[0]
|
||||
if plot is None:
|
||||
plot = plt.imshow(frame)
|
||||
else:
|
||||
plot.set_data(frame)
|
||||
plt.pause(0.01)
|
||||
|
||||
# check if the figure is closed, and if so, get out of the loop
|
||||
if not plt.fignum_exists(fig.number):
|
||||
break
|
||||
|
||||
plt.ioff() # interactive mode off
|
||||
plt.show()
|
||||
@@ -0,0 +1,23 @@
|
||||
## State Processors
|
||||
|
||||
This folder contains code to process states that are returned by the environment. This is a little bit different from the `processors` defined in the `pyrobolearn/processors` folder which can process the state data but let the state unchanged. That is, it doesn't change the state data, thus if you have a policy and a value function (which have and use the same state as input), each of these approximators can use their own processors on the state without one set of processors affecting the state data. This can lead to some overhead in processing time if the same processors have to be used for the various approximators / controllers. In contrast, the processors defined here wraps the states (i.e. they are also considered as `State` as they inherit from it), and the modified data will be apparent to all the approximators / controllers that take as inputs the state. Wrapping the states allows also to use different processors for different states, and inheriting from `State` allows to use the various operations defined there. For the latter case, you can for instance add two state processors.
|
||||
|
||||
Here is a pseudo-code to illustrate the above points:
|
||||
```python
|
||||
s1 = State1(args)
|
||||
s2 = State2(args)
|
||||
|
||||
s = s1 + s2
|
||||
proc_state_1 = StateProcessor(s, args)
|
||||
proc_state_2 = StateProcessor1(s1, args1) + StateProcessor2(s2, args2)
|
||||
proc_state_3 = StateProcessor2(StateProcessor1(s, args1), args2)
|
||||
|
||||
# to compute the states, just call them which would normally provide different data
|
||||
state_data = s()
|
||||
proc_state_1_data = proc_state_1()
|
||||
proc_state_2_data = proc_state_2()
|
||||
proc_state_3_data = proc_state_3()
|
||||
```
|
||||
|
||||
Important state processors are processors that center, normalize, standardize, and/or clip the state data.
|
||||
|
||||
@@ -129,6 +129,7 @@ class State(object):
|
||||
self._normalizer = None
|
||||
self._noiser = None # for noise
|
||||
self._name = name
|
||||
self._training_mode = False
|
||||
|
||||
# create ordered set which is useful if this state is a combination of multiple states
|
||||
self._states = OrderedSet()
|
||||
@@ -425,10 +426,23 @@ class State(object):
|
||||
# check if distribution is discrete/continuous
|
||||
pass
|
||||
|
||||
@property
|
||||
def in_training_mode(self):
|
||||
"""Return True if we are in training mode."""
|
||||
return self._training_mode
|
||||
|
||||
###########
|
||||
# Methods #
|
||||
###########
|
||||
|
||||
def train(self):
|
||||
"""Set the state in training mode."""
|
||||
self._training_mode = True
|
||||
|
||||
def eval(self):
|
||||
"""Set the state in evaluation / test mode."""
|
||||
self._training_mode = False
|
||||
|
||||
def is_combined_states(self):
|
||||
"""
|
||||
Return a boolean value depending if the state is a combination of states.
|
||||
|
||||
Reference in New Issue
Block a user