diff --git a/pyrobolearn/__init__.py b/pyrobolearn/__init__.py index b1f650a..5c57893 100644 --- a/pyrobolearn/__init__.py +++ b/pyrobolearn/__init__.py @@ -53,7 +53,7 @@ import tasks # import metrics # import optimizers -# import optim +# import optimizers # import algos import algos diff --git a/pyrobolearn/actions/README.md b/pyrobolearn/actions/README.md index f19a5a7..1546b3c 100644 --- a/pyrobolearn/actions/README.md +++ b/pyrobolearn/actions/README.md @@ -2,9 +2,13 @@ The `Action` is produced by the policy in response to a certain state/observation. From a programming point of view, compared to the `State` class, the action is a setter object. Thus, they have a very close relationship and share many functionalities. Some actions are mutually exclusive and cannot be executed at the same time. -An action is defined as something that affects the environment; that forces the environment to go to the next state. For instance, an action could be the desired joint positions, but also an abstract action such as 'open a door' which would then open a door in the simulator and load the next part of the world. +An action is defined as something that affects the environment; that forces the environment to go to the next state. For instance, an action could be the desired joint positions, but also an abstract action such as 'open a door' which would then open a door in the simulator and load the next part of the world. This would depend on how the user implemented his/her action class. -In the framework, the `Action` class is decoupled from the policy and environment rendering it more modular [1]. Nevertheless, the `Action` class still acts as a bridge between the policy and environment. In addition to be the output of a policy/controller, it can also be the input to some value estimators, dynamic models, reward functions, and so on. +In the framework, the `Action` class is decoupled from the policy and environment rendering it more modular and [flexible](https://en.wikipedia.org/wiki/Composition_over_inheritance). Nevertheless, the `Action` class still acts as a bridge between the policy and environment. In addition to be the output of a policy/controller, it can also be the input to some value estimators, dynamic models, reward functions, and so on. -References: - [1] "Wikipedia: Composition over Inheritance", https://en.wikipedia.org/wiki/Composition_over_inheritance \ No newline at end of file +You can for instance call `JointPositionAction(robot)`, and this will use position control to set the given joint positions. By giving the robot as input to the `JointPositionAction` class, it will automatically get the number of joints that the given robot possesses. This can then later be useful for instance when building a certain learning model for a policy. For example, assume we want to use a multilayer perceptron as the policy. The number of units on the last layer depends on the number of joints of the considered robot. Using the `MLPPolicy(outputs=actions)` it will automatically sets the correct number of output units depending on the considered robot. + + +#### What to check/look next? + +Check first the `states` folder if not already done, then the `approximators`, `policies`, `rewards`, and `envs` folders. diff --git a/pyrobolearn/algos/README.md b/pyrobolearn/algos/README.md new file mode 100644 index 0000000..1ab43b7 --- /dev/null +++ b/pyrobolearn/algos/README.md @@ -0,0 +1,12 @@ +## Algos + +This folder contains the various learning algorithms. Learning algorithms describe how to optimize the (hyper-)parameters of a particular learning model using a given loss function and optimizer. +Learning algorithms should not be confused with the models. + +They can be divided into two main categories: +* supervised / unsupervised learning algorithms: This type of algorithm describes how to update the (hyper-)parameters of a learning model given some input (and possibly output) data, a loss function to evaluate its performance, and an optimizer. +* reinforcement learning algorithms: + * In the model-free paradigm: the algorithm basically performs 3 main steps: + 1. Exploration: the algorithm describes how the policy should explore in the given environment and collect the various states, actions and rewards. + 2. Evaluation: it evaluates the actions taken by the policy using a certain estimator + 3. Update: this step is similar to supervised/unsupervised learning algorithms, where it updates the (hyper-)parameters of the various approximators (e.g. policies, value approximators, etc) using the given loss function and optimizer. diff --git a/pyrobolearn/approximators/README.md b/pyrobolearn/approximators/README.md new file mode 100644 index 0000000..cdb5d34 --- /dev/null +++ b/pyrobolearn/approximators/README.md @@ -0,0 +1,11 @@ +## Approximators + +This folder provides the various approximators used in the PRL framework. Approximators are a layer on top of the learning models and can accept as inputs/outputs the `State` and `Action` classes defined in this framework in addition to normal tensors. In contrast, the learning models are completely independent of the various classes defined in this framework and can thus be used in other projects. + +Approximators can be used to define: +- policies which map states to actions +- value function approximators which map states (and possibly actions) to a scalar. +- dynamic transition function approximators which map states and actions to the next state. This is notably useful in the model-based reinforcement learning paradigm. +- reward function approximator which map states/actions to a scalar. This is notably useful in the inverse reinforcement learning paradigm. + +Approximators can be trained / optimized using a loss function and an optimizer. \ No newline at end of file diff --git a/pyrobolearn/backends/README.md b/pyrobolearn/backends/README.md new file mode 100644 index 0000000..5e95153 --- /dev/null +++ b/pyrobolearn/backends/README.md @@ -0,0 +1,8 @@ +## Backends + +The general idea is to provide different backends such that different tensor frameworks can be used. These include for instance numpy (with autograd), pytorch, and tensorflow. +These frameworks use different data structures and different signatures for the methods. + +It would be nice to have learning models that are more or less independent of the tensor framework as done in Keras. + +This is mostly an idea that I had in a later stage, and thus is not operational for the moment. It would require to refactor a bit the code, as currently our code is coupled to the pytorch and numpy frameworks. diff --git a/pyrobolearn/control/README.md b/pyrobolearn/control/README.md new file mode 100644 index 0000000..0c5cef7 --- /dev/null +++ b/pyrobolearn/control/README.md @@ -0,0 +1,3 @@ +## Control processes/algorithms + +This folder will contain in the future control processes/algorithms. diff --git a/pyrobolearn/controllers/README.md b/pyrobolearn/controllers/README.md new file mode 100644 index 0000000..8bea6ee --- /dev/null +++ b/pyrobolearn/controllers/README.md @@ -0,0 +1,7 @@ +## Controllers + +Controllers are basically policies that do not possess any (hyper-)parameters to optimize. They are manually coded by the user. + +Planning (TODO): +- [ ] add some finite state machines +- [ ] add some classical locomotion controllers (based on ZMP, optimization, etc) diff --git a/pyrobolearn/dynamics/README.md b/pyrobolearn/dynamics/README.md new file mode 100644 index 0000000..f2f79a2 --- /dev/null +++ b/pyrobolearn/dynamics/README.md @@ -0,0 +1,5 @@ +## Dynamic models + +Dynamic models are models that given the current state (or a history of states), and the current action compute the next state. If dynamic models are provided or trained, we are often in a model-based reinforcement learning (also known as optimal control) paradigm. + +Dynamic models will be soon added to the framework. diff --git a/pyrobolearn/envs/README.md b/pyrobolearn/envs/README.md new file mode 100644 index 0000000..9a4c20e --- /dev/null +++ b/pyrobolearn/envs/README.md @@ -0,0 +1,31 @@ +## Environments + +This folder provides the environment as it described in an imitation / reinforcement learning setting. That is, given an action performed by an agent (i.e. policy), the environment computes and returns the next state and possibly a reward. A policy can then interact with this environment, and be trained. + +The environment defines the world, the rewards, and the states that are returned by it. To allow our framework to be flexible, the world, rewards, states, and terminal conditions are decoupled from the environment, and defined outside of this one and then provided as inputs to the `Env` class. See the `worlds`, `states`, and `rewards` folders. We thus favor [composition over inheritance](https://en.wikipedia.org/wiki/Composition_over_inheritance). This is a different approach compared to what is usually done with the [OpenAI gym](https://github.com/openai/gym) framework, where a world, state, rewards are defined inside the class that inherits from `gym.Env`. + + +```python +from itertools import count +import time +import pyrobolearn as prl + + +sim = prl.simulators.BulletSim() +world = prl.worlds.BasicWorld(sim) +robot = prl.robots.loadRobot('wam') # try another robot such as 'coman' or 'littledog' + +state = prl.states.JointPositionState(robot) + prl.states.JointVelocityState(robot) # you can add other states +reward = prl.rewards.() # define the reward/cost +env = prl.envs.Env(world, state, reward, terminal_condition=None) # the state, world, reward, and terminal condition are defined outside the environment + +for t in count(): + next_obs, rew, done, info = env.step() # this will ask the state to read or compute the next value, and perform a step in the simulator + time.sleep(1./240) + if (t % 240) == 0: + print(reward) # print the reward +``` + +#### What to check/look next? + +Check first the `policies` and `tasks` folders. diff --git a/pyrobolearn/experiments/README.md b/pyrobolearn/experiments/README.md new file mode 100644 index 0000000..b4d5a14 --- /dev/null +++ b/pyrobolearn/experiments/README.md @@ -0,0 +1,16 @@ +## Experiments + +In this folder, we define the `Experiment` class which is the highest-level class of our framework. More specifically, it allows to organize which tasks to run, which metrics to use, and allows to easily compare different models, algos, methods, and so on. + +The idea would be to define everything (world, robot, states, actions, rewards, environment, policy, algorithm, etc) inside the class that inherits from the `Experiment` class, and we should be able to run it using few commands. +```python +experiment = MyExperiment() +print(experiment.description()) +experiment.train() +experiment.test() +experiment.plot() +``` + +This would I hope ease the reproduction of experiments performed in the literature. + +Experiments will be added in a later stage. diff --git a/pyrobolearn/filters/README.md b/pyrobolearn/filters/README.md new file mode 100644 index 0000000..2e7c40c --- /dev/null +++ b/pyrobolearn/filters/README.md @@ -0,0 +1,8 @@ +## Filters + +This folder provides different filters / state estimators. These include for the moment: +- Histogram filter +- Kalman filter +- Extended Kalman filter +- Unscented Kalman filter +- Particle Filter diff --git a/pyrobolearn/filters/description.txt b/pyrobolearn/filters/description.txt deleted file mode 100644 index 540ec8f..0000000 --- a/pyrobolearn/filters/description.txt +++ /dev/null @@ -1 +0,0 @@ -filters --> state estimators diff --git a/pyrobolearn/metrics/README.md b/pyrobolearn/metrics/README.md new file mode 100644 index 0000000..eab6148 --- /dev/null +++ b/pyrobolearn/metrics/README.md @@ -0,0 +1,6 @@ +## Metrics + +Different learning tasks use different metrics. For instance, in transfer learning other metrics are used to evaluate the task than in reinforcement learning. + +This folder will contain in the future the various metrics. They should be able to collect various information, evaluate how well the learning task is performed, and plot the results. +By providing the different metrics, the user can select which metric he/she wants to use to evaluate his/her learning task. diff --git a/pyrobolearn/models/README.md b/pyrobolearn/models/README.md index cd6b020..817b6ff 100644 --- a/pyrobolearn/models/README.md +++ b/pyrobolearn/models/README.md @@ -1,3 +1,25 @@ ## Learning models -In this folder, we provide the various learning models. These can be categorized into two categories: movement primitives and general function approximators. \ No newline at end of file +In this folder, we provide the various learning models. These can be categorized into two categories: movement primitives and general function approximators. + +These include: +- Central pattern generators (CPG; the version provided by ) +- Dynamic movement primitives (DMP) +- Probabilistic movement primitives (ProMP) +- Kernelized movement primitives (KMP) +- Linear models +- PCA models +- Polynomial models +- Gaussian mixture models (GMM) with its regression counterpart (GMR) +- Gaussian processes (GP; it uses/wraps the [GPyTorch](https://github.com/cornellius-gp/gpytorch) library) +- Neural networks (currently, only MLP are provided) + +TODO: +- [ ] finish to implement the models +- [ ] provide multiple tests/examples for each model +- [ ] implement other models such as HMMs + + +#### what to check/look next? + +Check the `approximators`, `policies`, `values`, and `dynamics` folders. diff --git a/pyrobolearn/optim/optimizer.py b/pyrobolearn/optim/optimizer.py deleted file mode 100644 index 3ce00e5..0000000 --- a/pyrobolearn/optim/optimizer.py +++ /dev/null @@ -1,786 +0,0 @@ -#!/usr/bin/env python -"""Provide various optimizers. - -Optimizers allows to optimize (i.e. minimize or maximize) a utility function (also known as objective function, -fitness function, loss, etc.) with or without constraints and bounds. Notably, it assumes the functions have some -parameters that the optimizer can update. - -Mathematically, this is described as: - -.. math:: \min_{x \in R^n} f(x) - -subject to - -.. math:: - - g_i(x) \geq 0, \quad i = 1,...,m - h_j(x) = 0, \quad j = 1,...,p - x_l \leq x \leq x_u - -For trajectory optimization, check "An Introduction to Trajectory Optimization: How to do your own Direct Collocation". -""" - -# TODO: trajectory optimization - -from abc import ABCMeta, abstractmethod - -# Numpy with autograd -import autograd.numpy as np # Thinly-wrapped numpy -from autograd import grad # The only autograd function you may ever need - -# Scipy optimizer -import scipy - -# Pytorch optimizers -import torch.nn as nn -import torch.optim as optim - -# NLopt optimizers -try: - import nlopt -except ImportError as e: - raise ImportError(e.__str__() + "\n HINT: you can install nlopt via `pip install nlopt`.") - -# IPopt optimizer -try: - import ipopt -except ImportError as e: - raise ImportError(e.__str__() + "\n HINT: you can install ipopt using the `pyrobolearn/scripts/install_ipopt.sh`." - "If ipopt is already installed, you can install the python wrapper via " - "`pip install ipopt`.") - -# CVXOPT -# import cvxopt -# CVXPY: nice wrapper around cvxopt -# import cvxpy -# Quadprog -# import quadprog - -# QPsolvers optimizers: unified Python interface for multiple QP solvers (cvxopt, cvxpy, quadprog,...) -try: - import qpsolvers -except ImportError as e: - raise ImportError(e.__str__() + "\n HINT: you can install qpsolvers directly via 'pip install qpsolvers'.") - -# Bayesian optimization -try: - import GPy - import GPyOpt -except ImportError as e: - raise ImportError(e.__str__() + "\n HINT: you can install GPy/GPyOpt directly via 'pip install GPy' and " - "'pip install GPyOpt'.") - -# CMA-ES -try: - import cma -except ImportError as e: - raise ImportError(e.__str__() + "\n HINT: you can install CMA-ES or `pycma` directly via 'pip install cma'.") - - -__author__ = "Brian Delhaisse" -__copyright__ = "Copyright 2018, PyRoboLearn" -__credits__ = ["Brian Delhaisse"] -__license__ = "MIT" -__version__ = "1.0.0" -__maintainer__ = "Brian Delhaisse" -__email__ = "briandelhaisse@gmail.com" -__status__ = "Development" - - -################################################################## -# OPTIMIZER # -################################################################## - - -class Optimizer(object): - r"""Optimizer abstract class - - This is an abstract class from which all optimizers inherit from. Most of the child optimizer classes are wrappers - around the original optimizer. This is to provide the same common interface to all the optimizers, and convert - seamlessly to the correct data types. - - Optimizers are often given as a parameter to the learning algorithms, but can also be used of out of the box - directly on models. Several original optimizers can be given to the learning algorithm which will automatically - wrap the optimizer with the corresponding wrapper to provide a common interface. - - In their most natural form, optimizers are used to ... optimization process which can be described mathematically - by: - - .. math:: - - \min_{\theta} J(\theta) \mbox{ subj. to constraints} - - \max_{\theta} J(\theta) - - Several optimizers provides - - The list of optimizers available are from the following libraries: - * nlopt - * ipopt - * torch.optim - * GPy / GPyOpt - * scipy.optimize - * qpsolvers (which includes cvxpy) - * cmaes - * pso - - Each one of them expect a certain kind of type of the parameters. - - Optimizers can be divided into: - * global vs local - * derivative-free vs gradient-based - * without constraints vs with (equality and/or inequality) constraints - - Many implemented optimizers that can be found online are specific to a certain type of learning model. - If necessary, a conversion or wrapping process is carried out to make the optimizer work with the given learning - model. - """ - __metaclass__ = ABCMeta - - def __init__(self, model, losses, hyperparameters): - """ - - :param model: a certain type of model. If the original model is given instead of an instance of `Model`, then - the model will be wrapped appropriately. - :param losses: - :param hyperparameters: - """ - pass - - -################################################################## -# SCIPY # -################################################################## - -class Scipy(object): - r"""Scipy optimizer - - This uses the `scipy.optimize.minimize` to optimize a given objective function under various bounds and - constraints. Specifically, it consists of the minimization of a scalar function of one or more variables. - In general, the optimization problems are of the form: - - .. math:: - - \min_{x \in R^n} f(x) - - subject to - - .. math:: - - g_i(x) \geq 0, \quad i = 1,...,m - h_j(x) = 0, \quad j = 1,...,p - - where :math:`x` is a vector of one or more variables, :math:`g_i(x)` are the inequality constraints, and - :math:`h_j(x)` are the equality constrains. - - Optionally, the lower and upper bounds for each element in :math:`x` can also be specified using the `bounds` - argument. - - Several methods/optimizers are available: - - - - Note that only 'COBYLA' and 'SLSQP' support constraints, where the former only supports inequality constraints. - - References: - [1] scipy.optimize.minimize: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html - """ - - def __init__(self, method='SLSQP'): - """ - Initialize the scipy method - - Args: - method (str, callable): - - 'Nelder-Mead' :ref:`(see here) ` - - 'Powell' :ref:`(see here) ` - - 'CG' :ref:`(see here) ` - - 'BFGS' :ref:`(see here) ` - - 'Newton-CG' :ref:`(see here) ` - - 'L-BFGS-B' :ref:`(see here) ` - - 'TNC' :ref:`(see here) ` - - 'COBYLA' :ref:`(see here) ` - - 'SLSQP' :ref:`(see here) ` - - 'dogleg' :ref:`(see here) ` - - 'trust-ncg' :ref:`(see here) ` - - custom - a callable object (added in version 0.14.0), - - """ - # define optimization method - # By default, it will be 'BFGS', 'L-BFGS-B', or 'SLSQP' depending on the constraints and bounds - # If constraints, it can only be 'COBYLA' or 'SLSQP'. COBYLA only supports inequality constraints. - self.method = method - - def optimize(self, maxiter=1e6, verbose=True): - # define objective function to MINIMIZE - # f = lambda x: -(x.T.dot(C)).dot(x) - def f(x): - return -(x.T.dot(C)).dot(x) - - # define initial guess - x0 = np.ones((M,)) # np.zeros((M,)) - - # define 1st constraints: norm of 1 - constraints = [{'type': 'eq', 'fun': lambda x: x.T.dot(x) - 1, 'jac': None, 'args': ()}] - - # define bounds: each vector u have a norm of 1 thus each parameter is between -1 and 1 - bounds = [(-1., 1.)] * M - - # optimize recursively - evals, evecs = [], [] - messages = {} - options = {'maxiter': maxiter, 'disp': verbose} - for i in range(M): - if i != 0: - # add orthogonality constraint - constraints.append({'type': 'eq', 'fun': lambda u: u1.T.dot(u)}) - - # minimize --> it returns an instance of OptimizeResult - result = scipy.optimize.minimize(f, x0, args=(), method=self.method, jac=None, hess=None, bounds=bounds, - constraints=constraints, tol=None, callback=None, options=options) - - print(result.success) - print(result.message) - print(result.fun) - print(result.x) - - -################################################################## -# Quadratic Programming # -################################################################## - -# class CVXOPT(Optimizer): -# r"""Convex Optimizer -# -# Note: cvxpy module is a nice wrapper around cvxopt that follows paradigm of a disciplined convex programming. -# -# References: -# [1] Python Software for Convex Optimization: https://cvxopt.org/ -# [2] Github repo: https://github.com/cvxopt/cvxopt -# """ -# pass -# -# -# class CVXPY(Optimizer): -# r"""Convex Optimizer -# -# References: -# [1] CVXPY: http://www.cvxpy.org/ -# [2] Github repo: https://github.com/cvxgrp/cvxpy -# """ -# pass -# -# -# class QuadProg(object): -# r"""Quadprog -# -# References: -# [1] Github repo: https://github.com/rmcgibbo/quadprog -# """ -# pass - -class QP(object): - r"""Quadratic Programming solvers - - This class uses the `qpsolvers` which is a unified Python interface for multiple QP solvers [1,2]. - - .. math:: - - \min_{x \in R^n} \frac{1}{2} x^T P x + q^T x - - subject to - - .. math:: - - Gx \leq h - Ax = b - - where :math:`x` is the vector of optimization variables, the matrix :math:`P` and vector :math:`q` are used to - define any quadratic objective function on these variables, while the matrix-vector couples :math:`(G,h)` and - :math:`(A,b)` respectively define inequality and equality constraints. Vector inequalities apply coordinate by - coordinate [1]. - - - Dense solvers: - - CVXOPT - - CVXPY - - qpOASES - - quadprog - - Sparse solvers: - - ECOS as wrapped by CVXPY - - Gurobi - - MOSEK - - OSQP - - Check the available solvers by calling `print(qpsolvers.available_solvers)`. - - Notes: Many solvers (including CVXOPT, OSQP and quadprog) assume that `P` is a symmetric matrix, and may return - erroneous results when that is not the case. You can set ``sym_proj=True`` to project `P` on its symmetric part, - at the cost of some computation time. - - References: - [1] QP in Python: https://scaron.info/blog/quadratic-programming-in-python.html - [2] Github repo: https://github.com/stephane-caron/qpsolvers - """ - - def __init__(self, method='quadprog'): - """ - Initialize the QP solver. - - Args: - method (str): ['cvxopt', 'cvxpy', 'ecos', 'gurobi', 'mosek', 'osqp', 'qpoases', 'quadprog'] - """ - solvers = set(qpsolvers.available_solvers) - if len(solvers) == 0: - raise ValueError("No QP solvers have been found on this computer. Please install one of the QP modules") - if method not in solvers: - method = 'quadprog' - self.method = method - - # check methods that require a symmetric matrix for P - methods = ['cvxopt', 'osqp', 'quadprog'] - self.sym_proj = True if self.method in set(methods) else False - - def is_symmetric(self, X, tol=1e-8): - return np.allclose(X, X.T, atol=tol) - - def optimize(self, P, q, x0=None, G=None, h=None, A=None, b=None): - return qpsolvers.solve_qp(P, q, G, h, A, b, solver=self.method, initvals=x0, sym_proj=self.sym_proj) - - -################################################################## -# NLOPT # -################################################################## - -class NLopt(object): - r"""Non-Linear Optimizer - - Non-linear optimizers based on the `nlopt` libraries. - - Here is a brief of lists of the current algorithms implemented: - * - - Nonlinear optimization algos that can handle nonlinear inequality and EQUALITY constraints are: - - ISRES (Improved Stochastic Ranking Evolution Strategy) --> global derivative-free - - COBYLA (Constrained Optimization BY Linear Approximations) --> local derivative-free - - SLSQP (Sequential Least-SQuares Programming) --> local gradient-based - - AUGLAG (AUGmented LAGrangian) --> global/local derivative-free/gradient based (determined based on the - subsidiary algo) - - More information about: - - algorithms: https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/ - - References: - [1] NLopt: https://nlopt.readthedocs.io/en/latest/ - [2] NLopt with Python: with Python: https://nlopt.readthedocs.io/en/latest/NLopt_Python_Reference/ - [3] Github repo: https://github.com/stevengj/nlopt - """ - - # def __init__(self, model, losses, hyperparameters, seed): - # super(NLopt, self).__init__(model, losses, hyperparameters) - - def __init__(self, method, submethod=None, seed=None): - - # define useful variables - self.results = {1: 'success', 2: 'stop_val reached', 3: 'ftol reached', 4: 'xtol reached', - 5: 'maxeval reached', 6: 'maxtime reached', -1: 'failure', -2: 'invalid args', - -3: 'out of memory', -4: 'roundoff limited', -5: 'forced stop'} - - # define random seed - nlopt.srand(seed) - - # define which solver to use - def get_opt(method): - if method == 'ISRES': - return nlopt.opt(nlopt.GN_ISRES, M) - elif method == 'COBYLA': - return nlopt.opt(nlopt.LN_COBYLA, M) - elif method == 'SLSQP': - return nlopt.opt(nlopt.LD_SLSQP, M) - elif method == 'AUGLAG': - return nlopt.opt(nlopt.AUGLAG, M) - else: - raise NotImplementedError("The given method has not been implemented") - - if method is None: - method = 'SLSQP' - self.opt = get_opt(method) - - # define subsolver to use (if we use the AUGLAG method) - if method == 'AUGLAG': - if submethod is None: - submethod = 'SLSQP' - elif submethod == 'AUGLAG': - raise ValueError("Submethod should be different from AUGLAG") - subopt = get_opt(submethod) - subopt.set_lower_bounds(-1) - subopt.set_upper_bounds(1) - # subopt.set_ftol_rel(1e-2) - # subopt.set_maxeval(100) - self.opt.set_local_optimizer(subopt) - - def optimize(self): - # define objective function and its gradient - def f(x, grad): - if grad.size > 0: - grad[:] = 2 * x.T.dot(C) - return x.T.dot(C).dot(x) - - # define objective function to maximize - self.opt.set_max_objective(f) - - # if nlopt.GN_ISRES, we can define the population size - self.opt.set_population(0) # by default for ISRES: pop=20*(M+1) - - # define bound constraints (should be between -1 and 1 because the norm should be 1) - self.opt.set_lower_bounds(-1.) - self.opt.set_upper_bounds(1.) - - # define norm constraint and its gradient - def c1(x, grad): - if grad.size > 0: - grad[:] = 2 * x - return (x.T.dot(x) - 1) - - # define orthogonal constraint - class OrthogonalConstraint(object): - - def __init__(self, v): - self.v = np.copy(v) - - def constraint(self, x, grad): - if grad.size > 0: - grad[:] = self.v - return (x.T.dot(self.v)) - - # define equality constraints - opt.add_equality_constraint(c1, 0) - # opt.add_equality_mconstraint(constraints, tol) - - # define stopping criteria - # opt.set_stopval(stopval) - opt.set_ftol_rel(1e-8) - # opt.set_xtol_rel(1e-4) - opt.set_maxeval(100000) # nb of iteration - opt.set_maxtime(2) # time in secs - - # define initial value - x0 = np.array([0.1] * M) # important that the initial value != 0 for the computation of the grad! - - evals, evecs, msgs = [], [], {} - for i in range(M): - # add constraint - if i > 0: - c = OrthogonalConstraint(x) - opt.add_equality_constraint(c.constraint, 0) - - # optimize - try: - x = opt.optimize(x0) - except nlopt.RoundoffLimited as e: - pass - - # save values - evecs.append(x) # param vector - evals.append(opt.last_optimum_value()) # max value - msgs[i] = nlopt_results[opt.last_optimize_result()] - - -################################################################## -# IPOPT # -################################################################## - -class NormConstraint(object): - - def __init__(self): - pass - - def constraint(self, x): - return x.T.dot(x) - - def jacobian(self, x): - return 2 * x - - -class OrthogonalConstraint(object): - - def __init__(self, v): - self.v = np.copy(v) - - def constraint(self, x): - return x.T.dot(self.v) - - def jacobian(self, x): - return self.v - - -class _IPopt(object): - - def __init__(self, verbose=True): - self.verbose = verbose - self.iter_count = 0 - self.constraints = [] - - def add_constraint(self, constraint): - self.constraints.append(constraint) - - def objective(self, x): - # objective fct to minimize - return -x.T.dot(C).dot(x) - - def gradient(self, x): - # grad of the objective fct - return -2 * x.T.dot(C) - - def constraints(self, x): - return np.array([c.constraint(x) for c in self.constraints]) - - def jacobian(self, x): - return np.array([c.jacobian(x) for c in self.constraints]) - - # def hessian(self, x): - # pass - - def intermediate(self, alg_mod, iter_count, obj_value, inf_pr, inf_du, mu, d_norm, - regularization_size, alpha_du, alpha_pr, ls_trials): - if self.verbose: - print("Objective value at iteration #%d: %g" % (iter_count, obj_value)) - self.iter_count = iter_count - - -class IPopt(Optimizer): - r"""Interior-Point optimizer - - This is a wrapper around the `ipopt` library. It can be used to solve general nonlinear programming problems of - the form: - - .. math:: - - \min_{x \in R^n} f(x) - - subject to - - .. math:: - - g_L \leq g(x) \leq g_U - - x_L \leq x \leq x_U - - where :math:`x` are the optimization variables (possibly with upper an lower bounds, :math:`x_U` and :math:`x_L` - respectively), :math:`f(x)` is the objective function and :math:`g(x)` are the general nonlinear constraints. - The constraints, :math:`g(x)`, have lower and upper bounds. Note that equality constraints can be specified - by setting :math:`g^i_L = g^i_U`. - - More info: - - Check the documentation of the `ipopt.problem` method - - References: - [1] "On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear - programming", Wachter and Biegler, 2004 - [2] Ipopt: https://projects.coin-or.org/Ipopt - [3] Ipopt in Python: https://pythonhosted.org/ipopt/ - [4] Repos: https://github.com/coin-or/Ipopt and https://pypi.org/project/ipopt/ - """ - - def __init__(self, model, losses, hyperparameters): - super(IPopt, self).__init__(model, losses, hyperparameters) - - def optimize(self): - # define initial value - x0 = np.array([0.1] * N) # important that the initial value != 0 for the computation of the grad! - - # define (lower and upper) bound constraints - lb = [-1] * N - ub = [1] * N - - # define constraints; if upper and lower constraints (resp. cu and cl) are equal then equality constraint - cl = [1] + [0] * (N - 1) - cu = [1] + [0] * (N - 1) - - # create ipopt (which contains the objective function, its gradients, and constraints) - opt = _IPopt(verbose=False) - opt.add_constraint(NormConstraint()) - opt.add_constraint(OrthogonalConstraint(x)) - - # define the nonlinear optimization problem - nlp = ipopt.problem(n=N, m=len(cl[:i]), problem_obj=opt, lb=lb, ub=ub, cl=cl[:i], cu=cu[:i]) - - # solve problem - x, info = nlp.solve(x0) - - return x - - -################################################################## -# PYTORCH OPTIMIZERS # -################################################################## - - -class PyTorchOpt(Optimizer): - r"""PyTorch Optimizers - - This is a wrapper around the optimizers from pytorch. - """ - - def __init__(self, model, losses, hyperparameters): - super(PyTorchOpt, self).__init__(model, losses, hyperparameters) - - def add_constraint(self): - # it will add a constraint as the augmented lagrangian - pass - - -class Adam(object): - r"""Adam Optimizer - - References: - [1] "Adam: A Method for Stochastic Optimization", Kingma et al., 2014 - """ - - def __init__(self, learning_rate=1e-3, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, - max_grad_norm=None): # 0.5 - self.optimizer = None - self.learning_rate = learning_rate - self.betas = betas - self.eps = eps - self.weight_decay = weight_decay - self.amsgrad = amsgrad - self.max_grad_norm = max_grad_norm - - def reset(self): - self.optimizer = None - - def optimize(self, params, loss): - # create optimizer if necessary - if self.optimizer is None: - self.optimizer = optim.Adam(params, lr=self.learning_rate, betas=self.betas, eps=self.eps, - weight_decay=self.weight_decay, amsgrad=self.amsgrad) - - # optimize - self.optimizer.zero_grad() - loss.backward() - if self.max_grad_norm is not None: - nn.utils.clip_grad_norm_(params, self.max_grad_norm) - self.optimizer.step() - - -class Adadelta(object): - r"""Adadelta Optimizer - - References: - [1] "ADADELTA: An Adaptive Learning Rate Method", Zeiler, 2012 - """ - - def __init__(self, learning_rate=1., rho=0.9, eps=1e-6, weight_decay=0, max_grad_norm=None): #0.5 - self.optimizer = None - self.learning_rate = learning_rate - self.rho = rho - self.eps = eps - self.weight_decay = weight_decay - self.max_grad_norm = max_grad_norm - - def optimize(self, params, loss): - if self.optimizer is None: - self.optimizer = optim.Adadelta(params, lr=self.learning_rate, rho=self.rho, eps=self.eps, - weight_decay=self.weight_decay) - - # optimize - self.optimizer.zero_grad() - loss.backward() - if self.max_grad_norm is not None: - nn.utils.clip_grad_norm_(params, self.max_grad_norm) - self.optimizer.step() - - -class Adagrad(object): - r"""Adagrad Optimizer - - References: - [1] "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization", Duchi et al., 2011 - """ - - def __init__(self, learning_rate=0.01, learning_rate_decay=0, weight_decay=0, initial_accumumaltor_value=0, - max_grad_norm=None): # 0.5 - self.optimizer = None - self.learning_rate = learning_rate - self.learning_rate_decay = learning_rate_decay - self.weight_decay = weight_decay - self.initial_accumulator_value = initial_accumumaltor_value - self.max_grad_norm = max_grad_norm - - def optimize(self, params, loss): - if self.optimizer is None: - self.optimizer = optim.Adagrad(params, lr=self.learning_rate, lr_decay=self.learning_rate_decay, - weight_decay=self.weight_decay, - initial_accumulator_value=self.initial_accumulator_value) - - # optimize - self.optimizer.zero_grad() - loss.backward() - if self.max_grad_norm is not None: - nn.utils.clip_grad_norm_(params, self.max_grad_norm) - self.optimizer.step() - - -class RMSprop(object): - r"""RMSprop - - References: - [1] "RMSprop: Divide the gradient by a running average of its recent magnitude" (lecture 6.5), Tieleman and - Hinton, 2012 - [2] "Generating Sequences With Recurrent Neural Networks", Graves, 2014 - """ - - def __init__(self, learning_rate=1e-2, alpha=0.99, eps=1e-8, weight_decay=0, momentum=0, centered=False, - max_grad_norm=None): # 0.5 - self.optimizer = None - self.learning_rate = learning_rate - self.alpha = alpha - self.eps = eps - self.weight_decay = weight_decay - self.momentum = momentum - self.centered = centered - self.max_grad_norm = max_grad_norm - - def optimize(self, params, loss): - if self.optimizer is None: - self.optimizer = optim.RMSprop(params, lr=self.learning_rate, alpha=self.alpha, eps=self.eps, - weight_decay=self.weight_decay, momentum=self.momentum, - centered=self.centered) - - # optimize - self.optimizer.zero_grad() - loss.backward() - if self.max_grad_norm is not None: - nn.utils.clip_grad_norm_(params, self.max_grad_norm) - self.optimizer.step() - - -class SGD(object): - r"""Stochastic Gradient Descent - - References: - [1] "A Stochastic Approximation Method", Robbins and Monro, 1951 - [2] "On the importance of initialization and momentum in deep learning", Sutskever et al., 2013 - """ - - def __init__(self, learning_rate=1e-3, momentum=0, dampening=0, weight_decay=0, nesterov=False, - max_grad_norm=None): #0.5 - self.optimizer = None - self.learning_rate = learning_rate - self.momentum = momentum - self.dampening = dampening - self.weight_decay = weight_decay - self.nesterov = nesterov - self.max_grad_norm = max_grad_norm - - def optimize(self, params, loss): - # create optimizer if necessary - if self.optimizer is None: - self.optimizer = optim.SGD(params, lr=self.learning_rate, momentum=self.momentum, dampening=self.dampening, - weight_decay=self.weight_decay, nesterov=self.nesterov) - - # optimize - self.optimizer.zero_grad() - loss.backward() - if self.max_grad_norm is not None: - nn.utils.clip_grad_norm_(params, self.max_grad_norm) - self.optimizer.step() diff --git a/pyrobolearn/optimizers/README.md b/pyrobolearn/optimizers/README.md new file mode 100644 index 0000000..1a4db0b --- /dev/null +++ b/pyrobolearn/optimizers/README.md @@ -0,0 +1,18 @@ +## Optimizers + +This folder provides the various optimizers. It mostly wraps already existing optimizers but provides a common API. + +Optimizers allows to maximize/minimize a function with respect to its parameters/inputs under possibly various (equality and/or inequality) constraints with possibly different bounds on the parameters. + +Optimizers: +- [`scipy.optimize`](https://docs.scipy.org/doc/scipy/reference/optimize.html) +- [`torch.optim`](https://pytorch.org/docs/stable/optim.html) +- [`nlopt`](https://nlopt.readthedocs.io/en/latest/) +- [`ipopt`](https://projects.coin-or.org/Ipopt) +- [`qpsolvers`](https://github.com/stephane-caron/qpsolvers) +- [`cma`](https://github.com/CMA-ES/pycma) +- [`GPyOpt`](https://sheffieldml.github.io/GPyOpt/) + + +TODO: +- [ ] clean and finish to implement the various optimizers + structure the code in a better way diff --git a/pyrobolearn/optim/__init__.py b/pyrobolearn/optimizers/__init__.py similarity index 100% rename from pyrobolearn/optim/__init__.py rename to pyrobolearn/optimizers/__init__.py diff --git a/pyrobolearn/optim/cio.py b/pyrobolearn/optimizers/cio.py similarity index 99% rename from pyrobolearn/optim/cio.py rename to pyrobolearn/optimizers/cio.py index 10fdf24..65b78c5 100644 --- a/pyrobolearn/optim/cio.py +++ b/pyrobolearn/optimizers/cio.py @@ -62,4 +62,4 @@ class CIO(object): end_effector_quat = self.robot.getEndEffectorOrientations() def optimize(self): - pass \ No newline at end of file + pass diff --git a/pyrobolearn/optimizers/cma_optimizer.py b/pyrobolearn/optimizers/cma_optimizer.py new file mode 100644 index 0000000..d64e0cc --- /dev/null +++ b/pyrobolearn/optimizers/cma_optimizer.py @@ -0,0 +1,31 @@ +#!/usr/bin/env python +"""Provide a wrapper around the Covariance Adaptation Matrix (CMA) which uses an evolution strategy to optimize +the parameters. + +References: + [1] https://github.com/CMA-ES/pycma +""" + +import numpy as np + +# CMA-ES +try: + import cma +except ImportError as e: + raise ImportError(e.__str__() + "\n HINT: you can install CMA-ES or `pycma` directly via 'pip install cma'.") + +from optimizer import Optimizer + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +# TODO: check the `pyrobolearn/algos/cmaes` algo +class CMAES(object): + pass diff --git a/pyrobolearn/optimizers/gpyopt_optimizer.py b/pyrobolearn/optimizers/gpyopt_optimizer.py new file mode 100644 index 0000000..8926b07 --- /dev/null +++ b/pyrobolearn/optimizers/gpyopt_optimizer.py @@ -0,0 +1,32 @@ +#!/usr/bin/env python +"""Provide a wrapper around the GPyOpt optimizers for Bayesian Optimization (BO). + +References: + [1] https://sheffieldml.github.io/GPyOpt/ +""" + +import numpy as np + +# Bayesian optimization +try: + import GPy + import GPyOpt +except ImportError as e: + raise ImportError(e.__str__() + "\n HINT: you can install GPy/GPyOpt directly via 'pip install GPy' and " + "'pip install GPyOpt'.") + +from optimizer import Optimizer + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +# TODO: check the `pyrobolearn/algos/bo` algo +class BayesianOptimizer(object): + pass diff --git a/pyrobolearn/optimizers/ipopt_optimizer.py b/pyrobolearn/optimizers/ipopt_optimizer.py new file mode 100644 index 0000000..1958b07 --- /dev/null +++ b/pyrobolearn/optimizers/ipopt_optimizer.py @@ -0,0 +1,148 @@ +#!/usr/bin/env python +"""Provide a wrapper around the non-linear Interior Point optimizer (IPopt). + +References: + [1] https://projects.coin-or.org/Ipopt +""" + +import numpy as np + +# IPopt optimizer +try: + import ipopt +except ImportError as e: + raise ImportError(e.__str__() + "\n HINT: you can install ipopt using the `pyrobolearn/scripts/install_ipopt.sh`." + "If ipopt is already installed, you can install the python wrapper via " + "`pip install ipopt`.") + +from optimizer import Optimizer + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +class NormConstraint(object): + + def __init__(self): + pass + + def constraint(self, x): + return x.T.dot(x) + + def jacobian(self, x): + return 2 * x + + +class OrthogonalConstraint(object): + + def __init__(self, v): + self.v = np.copy(v) + + def constraint(self, x): + return x.T.dot(self.v) + + def jacobian(self, x): + return self.v + + +class _IPopt(object): + + def __init__(self, verbose=True): + self.verbose = verbose + self.iter_count = 0 + self.constraints = [] + + def add_constraint(self, constraint): + self.constraints.append(constraint) + + def objective(self, x): + # objective fct to minimize + return -x.T.dot(C).dot(x) + + def gradient(self, x): + # grad of the objective fct + return -2 * x.T.dot(C) + + def constraints(self, x): + return np.array([c.constraint(x) for c in self.constraints]) + + def jacobian(self, x): + return np.array([c.jacobian(x) for c in self.constraints]) + + # def hessian(self, x): + # pass + + def intermediate(self, alg_mod, iter_count, obj_value, inf_pr, inf_du, mu, d_norm, + regularization_size, alpha_du, alpha_pr, ls_trials): + if self.verbose: + print("Objective value at iteration #%d: %g" % (iter_count, obj_value)) + self.iter_count = iter_count + + +class IPopt(Optimizer): + r"""Interior-Point optimizer + + This is a wrapper around the `ipopt` library. It can be used to solve general nonlinear programming problems of + the form: + + .. math:: + + \min_{x \in R^n} f(x) + + subject to + + .. math:: + + g_L \leq g(x) \leq g_U + + x_L \leq x \leq x_U + + where :math:`x` are the optimization variables (possibly with upper an lower bounds, :math:`x_U` and :math:`x_L` + respectively), :math:`f(x)` is the objective function and :math:`g(x)` are the general nonlinear constraints. + The constraints, :math:`g(x)`, have lower and upper bounds. Note that equality constraints can be specified + by setting :math:`g^i_L = g^i_U`. + + More info: + - Check the documentation of the `ipopt.problem` method + + References: + [1] "On the implementation of an interior-point filter line-search algorithm for large-scale nonlinear + programming", Wachter and Biegler, 2004 + [2] Ipopt: https://projects.coin-or.org/Ipopt + [3] Ipopt in Python: https://pythonhosted.org/ipopt/ + [4] Repos: https://github.com/coin-or/Ipopt and https://pypi.org/project/ipopt/ + """ + + def __init__(self, model, losses, hyperparameters): + super(IPopt, self).__init__(model, losses, hyperparameters) + + def optimize(self): + # define initial value + x0 = np.array([0.1] * N) # important that the initial value != 0 for the computation of the grad! + + # define (lower and upper) bound constraints + lb = [-1] * N + ub = [1] * N + + # define constraints; if upper and lower constraints (resp. cu and cl) are equal then equality constraint + cl = [1] + [0] * (N - 1) + cu = [1] + [0] * (N - 1) + + # create ipopt (which contains the objective function, its gradients, and constraints) + opt = _IPopt(verbose=False) + opt.add_constraint(NormConstraint()) + opt.add_constraint(OrthogonalConstraint(x)) + + # define the nonlinear optimization problem + nlp = ipopt.problem(n=N, m=len(cl[:i]), problem_obj=opt, lb=lb, ub=ub, cl=cl[:i], cu=cu[:i]) + + # solve problem + x, info = nlp.solve(x0) + + return x diff --git a/pyrobolearn/optimizers/nlopt_optimizer.py b/pyrobolearn/optimizers/nlopt_optimizer.py new file mode 100644 index 0000000..2352fa2 --- /dev/null +++ b/pyrobolearn/optimizers/nlopt_optimizer.py @@ -0,0 +1,162 @@ +#!/usr/bin/env python +"""Provide a wrapper around the Non-Linear optimizers (NLopt). + +References: + [1] https://nlopt.readthedocs.io/en/latest/ +""" + +import numpy as np + +# NLopt optimizers +try: + import nlopt +except ImportError as e: + raise ImportError(e.__str__() + "\n HINT: you can install nlopt via `pip install nlopt`.") + +from optimizer import Optimizer + + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +class NLopt(object): + r"""Non-Linear Optimizer + + Non-linear optimizers based on the `nlopt` libraries. + + Here is a brief of lists of the current algorithms implemented: + * + + Nonlinear optimization algos that can handle nonlinear inequality and EQUALITY constraints are: + - ISRES (Improved Stochastic Ranking Evolution Strategy) --> global derivative-free + - COBYLA (Constrained Optimization BY Linear Approximations) --> local derivative-free + - SLSQP (Sequential Least-SQuares Programming) --> local gradient-based + - AUGLAG (AUGmented LAGrangian) --> global/local derivative-free/gradient based (determined based on the + subsidiary algo) + + More information about: + - algorithms: https://nlopt.readthedocs.io/en/latest/NLopt_Algorithms/ + + References: + [1] NLopt: https://nlopt.readthedocs.io/en/latest/ + [2] NLopt with Python: with Python: https://nlopt.readthedocs.io/en/latest/NLopt_Python_Reference/ + [3] Github repo: https://github.com/stevengj/nlopt + """ + + # def __init__(self, model, losses, hyperparameters, seed): + # super(NLopt, self).__init__(model, losses, hyperparameters) + + def __init__(self, method, submethod=None, seed=None): + + # define useful variables + self.results = {1: 'success', 2: 'stop_val reached', 3: 'ftol reached', 4: 'xtol reached', + 5: 'maxeval reached', 6: 'maxtime reached', -1: 'failure', -2: 'invalid args', + -3: 'out of memory', -4: 'roundoff limited', -5: 'forced stop'} + + # define random seed + nlopt.srand(seed) + + # define which solver to use + def get_opt(method): + if method == 'ISRES': + return nlopt.opt(nlopt.GN_ISRES, M) + elif method == 'COBYLA': + return nlopt.opt(nlopt.LN_COBYLA, M) + elif method == 'SLSQP': + return nlopt.opt(nlopt.LD_SLSQP, M) + elif method == 'AUGLAG': + return nlopt.opt(nlopt.AUGLAG, M) + else: + raise NotImplementedError("The given method has not been implemented") + + if method is None: + method = 'SLSQP' + self.opt = get_opt(method) + + # define subsolver to use (if we use the AUGLAG method) + if method == 'AUGLAG': + if submethod is None: + submethod = 'SLSQP' + elif submethod == 'AUGLAG': + raise ValueError("Submethod should be different from AUGLAG") + subopt = get_opt(submethod) + subopt.set_lower_bounds(-1) + subopt.set_upper_bounds(1) + # subopt.set_ftol_rel(1e-2) + # subopt.set_maxeval(100) + self.opt.set_local_optimizer(subopt) + + def optimize(self): + # define objective function and its gradient + def f(x, grad): + if grad.size > 0: + grad[:] = 2 * x.T.dot(C) + return x.T.dot(C).dot(x) + + # define objective function to maximize + self.opt.set_max_objective(f) + + # if nlopt.GN_ISRES, we can define the population size + self.opt.set_population(0) # by default for ISRES: pop=20*(M+1) + + # define bound constraints (should be between -1 and 1 because the norm should be 1) + self.opt.set_lower_bounds(-1.) + self.opt.set_upper_bounds(1.) + + # define norm constraint and its gradient + def c1(x, grad): + if grad.size > 0: + grad[:] = 2 * x + return (x.T.dot(x) - 1) + + # define orthogonal constraint + class OrthogonalConstraint(object): + + def __init__(self, v): + self.v = np.copy(v) + + def constraint(self, x, grad): + if grad.size > 0: + grad[:] = self.v + return (x.T.dot(self.v)) + + # define equality constraints + self.opt.add_equality_constraint(c1, 0) + # opt.add_equality_mconstraint(constraints, tol) + + # define stopping criteria + # self.opt.set_stopval(stopval) + self.opt.set_ftol_rel(1e-8) + # opt.set_xtol_rel(1e-4) + self.opt.set_maxeval(100000) # nb of iteration + self.opt.set_maxtime(2) # time in secs + + # define initial value + x0 = np.array([0.1] * M) # important that the initial value != 0 for the computation of the grad! + + evals, evecs, msgs = [], [], {} + for i in range(M): + # add constraint + if i > 0: + c = OrthogonalConstraint(x) + self.opt.add_equality_constraint(c.constraint, 0) + + # optimize + try: + x = self.opt.optimize(x0) + except nlopt.RoundoffLimited as e: + pass + + # save values + evecs.append(x) # param vector + evals.append(self.opt.last_optimum_value()) # max value + msgs[i] = nlopt_results[self.opt.last_optimize_result()] + + diff --git a/pyrobolearn/optimizers/optimizer.py b/pyrobolearn/optimizers/optimizer.py new file mode 100644 index 0000000..3fb67e9 --- /dev/null +++ b/pyrobolearn/optimizers/optimizer.py @@ -0,0 +1,146 @@ +#!/usr/bin/env python +"""Provide the abstract optimizer class. + +Optimizers allows to optimize (i.e. minimize or maximize) a utility function (also known as objective function, +fitness function, loss, etc.) with or without constraints and bounds. Notably, it assumes the functions have some +parameters that the optimizer can update. + +Mathematically, this is described as: + +.. math:: \min_{x \in R^n} f(x) + +subject to + +.. math:: + + g_i(x) \geq 0, \quad i = 1,...,m + h_j(x) = 0, \quad j = 1,...,p + x_l \leq x \leq x_u + +For trajectory optimization, check "An Introduction to Trajectory Optimization: How to do your own Direct Collocation". +""" + +# TODO: trajectory optimization + +from abc import ABCMeta, abstractmethod + +# Numpy with autograd +# import autograd.numpy as np # Thinly-wrapped numpy +# from autograd import grad # The only autograd function you may ever need +import numpy as np + + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +class Optimizer(object): + r"""Optimizer abstract class + + This is an abstract class from which all optimizers inherit from. Most of the child optimizer classes are wrappers + around the original optimizer. This is to provide the same common interface to all the optimizers, and convert + seamlessly to the correct data types. + + Optimizers are often given as a parameter to the learning algorithms, but can also be used of out of the box + directly on models. Several original optimizers can be given to the learning algorithm which will automatically + wrap the optimizer with the corresponding wrapper to provide a common interface. + + In their most natural form, optimizers are used to ... optimization process which can be described mathematically + by: + + .. math:: + + \min_{\theta} J(\theta) \mbox{ subj. to constraints} + + \max_{\theta} J(\theta) + + Several optimizers provides + + The list of optimizers available are from the following libraries: + * nlopt + * ipopt + * torch.optim + * GPy / GPyOpt + * scipy.optimize + * qpsolvers (which includes cvxpy) + * cmaes + * pso + + Each one of them expect a certain kind of type of the parameters. + + Optimizers can be divided into: + * global vs local + * derivative-free vs gradient-based + * without constraints vs with (equality and/or inequality) constraints + + Many implemented optimizers that can be found online are specific to a certain type of learning model. + If necessary, a conversion or wrapping process is carried out to make the optimizer work with the given learning + model. + """ + __metaclass__ = ABCMeta + + def __init__(self, *args, **kwargs): # model, losses, hyperparameters): + """ + Initialize the optimizer. + """ + self.best_parameters = None + self.best_result = None + self.is_maximizing = True + + ############## + # Properties # + ############## + + @property + def best_parameters(self): + return self._best_parameters + + @best_parameters.setter + def best_parameters(self, params): + self._best_parameters = params + + @property + def best_result(self): + return self._best_result + + @best_result.setter + def best_result(self, result): + self._best_result = result + + @property + def is_maximizing(self): + return self._is_maximizing + + @is_maximizing.setter + def is_maximizing(self, boolean): + self._is_maximizing = bool(boolean) + + @property + def is_minimizing(self): + return not self.is_maximizing + + ########### + # Methods # + ########### + + def optimize(self, *args, **kwargs): + pass + + ############# + # Operators # + ############# + + def __repr__(self): + return self.__class__.__name__ + + def __str__(self): + return self.__class__.__name__ + + def __call__(self, *args, **kwargs): + return self.optimize(*args, **kwargs) diff --git a/pyrobolearn/optimizers/qpsolvers_optimizer.py b/pyrobolearn/optimizers/qpsolvers_optimizer.py new file mode 100644 index 0000000..c21abae --- /dev/null +++ b/pyrobolearn/optimizers/qpsolvers_optimizer.py @@ -0,0 +1,130 @@ +#!/usr/bin/env python +"""Provide a wrapper around quadratic programming solvers. + +References: + [1] https://github.com/stephane-caron/qpsolvers +""" + +import numpy as np + +# CVXOPT +# import cvxopt +# CVXPY: nice wrapper around cvxopt +# import cvxpy +# Quadprog +# import quadprog + +# QPsolvers optimizers: unified Python interface for multiple QP solvers (cvxopt, cvxpy, quadprog,...) +try: + import qpsolvers +except ImportError as e: + raise ImportError(e.__str__() + "\n HINT: you can install qpsolvers directly via 'pip install qpsolvers'.") + +from optimizer import Optimizer + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +# class CVXOPT(Optimizer): +# r"""Convex Optimizer +# +# Note: cvxpy module is a nice wrapper around cvxopt that follows paradigm of a disciplined convex programming. +# +# References: +# [1] Python Software for Convex Optimization: https://cvxopt.org/ +# [2] Github repo: https://github.com/cvxopt/cvxopt +# """ +# pass +# +# +# class CVXPY(Optimizer): +# r"""Convex Optimizer +# +# References: +# [1] CVXPY: http://www.cvxpy.org/ +# [2] Github repo: https://github.com/cvxgrp/cvxpy +# """ +# pass +# +# +# class QuadProg(object): +# r"""Quadprog +# +# References: +# [1] Github repo: https://github.com/rmcgibbo/quadprog +# """ +# pass + +class QP(object): + r"""Quadratic Programming solvers + + This class uses the `qpsolvers` which is a unified Python interface for multiple QP solvers [1,2]. + + .. math:: + + \min_{x \in R^n} \frac{1}{2} x^T P x + q^T x + + subject to + + .. math:: + + Gx \leq h + Ax = b + + where :math:`x` is the vector of optimization variables, the matrix :math:`P` and vector :math:`q` are used to + define any quadratic objective function on these variables, while the matrix-vector couples :math:`(G,h)` and + :math:`(A,b)` respectively define inequality and equality constraints. Vector inequalities apply coordinate by + coordinate [1]. + + - Dense solvers: + - CVXOPT + - CVXPY + - qpOASES + - quadprog + - Sparse solvers: + - ECOS as wrapped by CVXPY + - Gurobi + - MOSEK + - OSQP + + Check the available solvers by calling `print(qpsolvers.available_solvers)`. + + Notes: Many solvers (including CVXOPT, OSQP and quadprog) assume that `P` is a symmetric matrix, and may return + erroneous results when that is not the case. You can set ``sym_proj=True`` to project `P` on its symmetric part, + at the cost of some computation time. + + References: + [1] QP in Python: https://scaron.info/blog/quadratic-programming-in-python.html + [2] Github repo: https://github.com/stephane-caron/qpsolvers + """ + + def __init__(self, method='quadprog'): + """ + Initialize the QP solver. + + Args: + method (str): ['cvxopt', 'cvxpy', 'ecos', 'gurobi', 'mosek', 'osqp', 'qpoases', 'quadprog'] + """ + solvers = set(qpsolvers.available_solvers) + if len(solvers) == 0: + raise ValueError("No QP solvers have been found on this computer. Please install one of the QP modules") + if method not in solvers: + method = 'quadprog' + self.method = method + + # check methods that require a symmetric matrix for P + methods = ['cvxopt', 'osqp', 'quadprog'] + self.sym_proj = True if self.method in set(methods) else False + + def is_symmetric(self, X, tol=1e-8): + return np.allclose(X, X.T, atol=tol) + + def optimize(self, P, q, x0=None, G=None, h=None, A=None, b=None): + return qpsolvers.solve_qp(P, q, G, h, A, b, solver=self.method, initvals=x0, sym_proj=self.sym_proj) diff --git a/pyrobolearn/optimizers/scipy_optimizer.py b/pyrobolearn/optimizers/scipy_optimizer.py new file mode 100644 index 0000000..61e4975 --- /dev/null +++ b/pyrobolearn/optimizers/scipy_optimizer.py @@ -0,0 +1,112 @@ +#!/usr/bin/env python +"""Provide a wrapper around the Scipy optimizers. + +References: + [1] https://docs.scipy.org/doc/scipy/reference/optimize.html +""" + +import numpy as np +import scipy + +from optimizer import Optimizer + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +class Scipy(object): + r"""Scipy optimizer + + This uses the `scipy.optimize.minimize` to optimize a given objective function under various bounds and + constraints. Specifically, it consists of the minimization of a scalar function of one or more variables. + In general, the optimization problems are of the form: + + .. math:: + + \min_{x \in R^n} f(x) + + subject to + + .. math:: + + g_i(x) \geq 0, \quad i = 1,...,m + h_j(x) = 0, \quad j = 1,...,p + + where :math:`x` is a vector of one or more variables, :math:`g_i(x)` are the inequality constraints, and + :math:`h_j(x)` are the equality constrains. + + Optionally, the lower and upper bounds for each element in :math:`x` can also be specified using the `bounds` + argument. + + Several methods/optimizers are available: + - + + Note that only 'COBYLA' and 'SLSQP' support constraints, where the former only supports inequality constraints. + + References: + [1] scipy.optimize.minimize: https://docs.scipy.org/doc/scipy/reference/generated/scipy.optimize.minimize.html + """ + + def __init__(self, method='SLSQP'): + """ + Initialize the scipy method + + Args: + method (str, callable): + - 'Nelder-Mead' :ref:`(see here) ` + - 'Powell' :ref:`(see here) ` + - 'CG' :ref:`(see here) ` + - 'BFGS' :ref:`(see here) ` + - 'Newton-CG' :ref:`(see here) ` + - 'L-BFGS-B' :ref:`(see here) ` + - 'TNC' :ref:`(see here) ` + - 'COBYLA' :ref:`(see here) ` + - 'SLSQP' :ref:`(see here) ` + - 'dogleg' :ref:`(see here) ` + - 'trust-ncg' :ref:`(see here) ` + - custom - a callable object (added in version 0.14.0), + + """ + # define optimization method + # By default, it will be 'BFGS', 'L-BFGS-B', or 'SLSQP' depending on the constraints and bounds + # If constraints, it can only be 'COBYLA' or 'SLSQP'. COBYLA only supports inequality constraints. + self.method = method + + def optimize(self, maxiter=1e6, verbose=True): + # define objective function to MINIMIZE + # f = lambda x: -(x.T.dot(C)).dot(x) + def f(x): + return -(x.T.dot(C)).dot(x) + + # define initial guess + x0 = np.ones((M,)) # np.zeros((M,)) + + # define 1st constraints: norm of 1 + constraints = [{'type': 'eq', 'fun': lambda x: x.T.dot(x) - 1, 'jac': None, 'args': ()}] + + # define bounds: each vector u have a norm of 1 thus each parameter is between -1 and 1 + bounds = [(-1., 1.)] * M + + # optimize recursively + evals, evecs = [], [] + messages = {} + options = {'maxiter': maxiter, 'disp': verbose} + for i in range(M): + if i != 0: + # add orthogonality constraint + constraints.append({'type': 'eq', 'fun': lambda u: u1.T.dot(u)}) + + # minimize --> it returns an instance of OptimizeResult + result = scipy.optimize.minimize(f, x0, args=(), method=self.method, jac=None, hess=None, bounds=bounds, + constraints=constraints, tol=None, callback=None, options=options) + + print(result.success) + print(result.message) + print(result.fun) + print(result.x) diff --git a/pyrobolearn/optimizers/torch_optimizer.py b/pyrobolearn/optimizers/torch_optimizer.py new file mode 100644 index 0000000..a387475 --- /dev/null +++ b/pyrobolearn/optimizers/torch_optimizer.py @@ -0,0 +1,197 @@ +#!/usr/bin/env python +"""Provide wrappers around the torch optimizers. + +Compared to the optimizers present in the PyTorch library [1], which require to pass the parameters of the model when +instantiating them, here we can pass the parameters at a later stage. + +References: + [1] https://pytorch.org/docs/stable/optim.html +""" + +# Pytorch optimizers +import torch.nn as nn +import torch.optim as optim + +from optimizer import Optimizer + + +__author__ = "Brian Delhaisse" +__copyright__ = "Copyright 2018, PyRoboLearn" +__credits__ = ["Brian Delhaisse"] +__license__ = "MIT" +__version__ = "1.0.0" +__maintainer__ = "Brian Delhaisse" +__email__ = "briandelhaisse@gmail.com" +__status__ = "Development" + + +# class PyTorchOpt(Optimizer): +# r"""PyTorch Optimizers +# +# This is a wrapper around the optimizers from pytorch. +# """ +# +# def __init__(self, model, losses, hyperparameters): +# super(PyTorchOpt, self).__init__(model, losses, hyperparameters) +# +# def add_constraint(self): +# # it will add a constraint as the augmented lagrangian +# pass + + +class Adam(object): + r"""Adam Optimizer + + References: + [1] "Adam: A Method for Stochastic Optimization", Kingma et al., 2014 + """ + + def __init__(self, learning_rate=1e-3, betas=(0.9, 0.999), eps=1e-08, weight_decay=0, amsgrad=False, + max_grad_norm=None): # 0.5 + self.optimizer = None + self.learning_rate = learning_rate + self.betas = betas + self.eps = eps + self.weight_decay = weight_decay + self.amsgrad = amsgrad + self.max_grad_norm = max_grad_norm + + def reset(self): + self.optimizer = None + + def optimize(self, params, loss): + # create optimizer if necessary + if self.optimizer is None: + self.optimizer = optim.Adam(params, lr=self.learning_rate, betas=self.betas, eps=self.eps, + weight_decay=self.weight_decay, amsgrad=self.amsgrad) + + # optimize + self.optimizer.zero_grad() + loss.backward() + if self.max_grad_norm is not None: + nn.utils.clip_grad_norm_(params, self.max_grad_norm) + self.optimizer.step() + + +class Adadelta(object): + r"""Adadelta Optimizer + + References: + [1] "ADADELTA: An Adaptive Learning Rate Method", Zeiler, 2012 + """ + + def __init__(self, learning_rate=1., rho=0.9, eps=1e-6, weight_decay=0, max_grad_norm=None): #0.5 + self.optimizer = None + self.learning_rate = learning_rate + self.rho = rho + self.eps = eps + self.weight_decay = weight_decay + self.max_grad_norm = max_grad_norm + + def optimize(self, params, loss): + if self.optimizer is None: + self.optimizer = optim.Adadelta(params, lr=self.learning_rate, rho=self.rho, eps=self.eps, + weight_decay=self.weight_decay) + + # optimize + self.optimizer.zero_grad() + loss.backward() + if self.max_grad_norm is not None: + nn.utils.clip_grad_norm_(params, self.max_grad_norm) + self.optimizer.step() + + +class Adagrad(object): + r"""Adagrad Optimizer + + References: + [1] "Adaptive Subgradient Methods for Online Learning and Stochastic Optimization", Duchi et al., 2011 + """ + + def __init__(self, learning_rate=0.01, learning_rate_decay=0, weight_decay=0, initial_accumumaltor_value=0, + max_grad_norm=None): # 0.5 + self.optimizer = None + self.learning_rate = learning_rate + self.learning_rate_decay = learning_rate_decay + self.weight_decay = weight_decay + self.initial_accumulator_value = initial_accumumaltor_value + self.max_grad_norm = max_grad_norm + + def optimize(self, params, loss): + if self.optimizer is None: + self.optimizer = optim.Adagrad(params, lr=self.learning_rate, lr_decay=self.learning_rate_decay, + weight_decay=self.weight_decay, + initial_accumulator_value=self.initial_accumulator_value) + + # optimize + self.optimizer.zero_grad() + loss.backward() + if self.max_grad_norm is not None: + nn.utils.clip_grad_norm_(params, self.max_grad_norm) + self.optimizer.step() + + +class RMSprop(object): + r"""RMSprop + + References: + [1] "RMSprop: Divide the gradient by a running average of its recent magnitude" (lecture 6.5), Tieleman and + Hinton, 2012 + [2] "Generating Sequences With Recurrent Neural Networks", Graves, 2014 + """ + + def __init__(self, learning_rate=1e-2, alpha=0.99, eps=1e-8, weight_decay=0, momentum=0, centered=False, + max_grad_norm=None): # 0.5 + self.optimizer = None + self.learning_rate = learning_rate + self.alpha = alpha + self.eps = eps + self.weight_decay = weight_decay + self.momentum = momentum + self.centered = centered + self.max_grad_norm = max_grad_norm + + def optimize(self, params, loss): + if self.optimizer is None: + self.optimizer = optim.RMSprop(params, lr=self.learning_rate, alpha=self.alpha, eps=self.eps, + weight_decay=self.weight_decay, momentum=self.momentum, + centered=self.centered) + + # optimize + self.optimizer.zero_grad() + loss.backward() + if self.max_grad_norm is not None: + nn.utils.clip_grad_norm_(params, self.max_grad_norm) + self.optimizer.step() + + +class SGD(object): + r"""Stochastic Gradient Descent + + References: + [1] "A Stochastic Approximation Method", Robbins and Monro, 1951 + [2] "On the importance of initialization and momentum in deep learning", Sutskever et al., 2013 + """ + + def __init__(self, learning_rate=1e-3, momentum=0, dampening=0, weight_decay=0, nesterov=False, + max_grad_norm=None): #0.5 + self.optimizer = None + self.learning_rate = learning_rate + self.momentum = momentum + self.dampening = dampening + self.weight_decay = weight_decay + self.nesterov = nesterov + self.max_grad_norm = max_grad_norm + + def optimize(self, params, loss): + # create optimizer if necessary + if self.optimizer is None: + self.optimizer = optim.SGD(params, lr=self.learning_rate, momentum=self.momentum, dampening=self.dampening, + weight_decay=self.weight_decay, nesterov=self.nesterov) + + # optimize + self.optimizer.zero_grad() + loss.backward() + if self.max_grad_norm is not None: + nn.utils.clip_grad_norm_(params, self.max_grad_norm) + self.optimizer.step() diff --git a/pyrobolearn/policies/README.md b/pyrobolearn/policies/README.md new file mode 100644 index 0000000..0775c67 --- /dev/null +++ b/pyrobolearn/policies/README.md @@ -0,0 +1,5 @@ +## Policies + +Policies in this framework are controllers that can be trained and map states to actions. They use directly the learning model or the `Approximator` class (which uses the learning model). + +In this framework, `State` and `Action` instances should be given to the `Policy` which would infer its input and output dimensions and build the model with the correct number of inputs/outputs. In contrast to the learning model, the policy should know how to feed the various input states to the inner learning model, such that if a picture and joint states are given to the policy it knows where to feed the corresponding input observations. diff --git a/pyrobolearn/processors/README.md b/pyrobolearn/processors/README.md new file mode 100644 index 0000000..c0319dc --- /dev/null +++ b/pyrobolearn/processors/README.md @@ -0,0 +1,7 @@ +## Processors + +Processors are functions that are applied to the inputs (respectively outputs) of an approximator/learning model before (respectively after) being processed by it. Processors might have parameters but these are not trainable/optimizable and are thus fixed and given at the beginning. + +These processors allow for instance to scale the input and/or output of a learning model. They accept as inputs a State/Action, numpy array, or torch Tensor. + +They can for instance be used to normalize the input state before it is fed to the policy. diff --git a/pyrobolearn/recorders/README.md b/pyrobolearn/recorders/README.md new file mode 100644 index 0000000..221600f --- /dev/null +++ b/pyrobolearn/recorders/README.md @@ -0,0 +1,3 @@ +## Recorders + +Recorders record the states and actions. diff --git a/pyrobolearn/rewards/README.md b/pyrobolearn/rewards/README.md index 1378bde..959b08e 100644 --- a/pyrobolearn/rewards/README.md +++ b/pyrobolearn/rewards/README.md @@ -1,3 +1,8 @@ ## Rewards -In this folder, we define the most common rewards used in reinforcement learning and optimization. \ No newline at end of file +In this folder, we define the most common rewards/costs used in reinforcement learning. + + +#### what to check/look next? + +Check the `envs` folder, which accepts the rewards. diff --git a/pyrobolearn/robots/README.md b/pyrobolearn/robots/README.md new file mode 100644 index 0000000..fe82655 --- /dev/null +++ b/pyrobolearn/robots/README.md @@ -0,0 +1,95 @@ +## Robots + +This folder contains the various robots that can be used in the PRL framework. Currently, they can be loaded in the [PyBullet](https://pybullet.org/wordpress/) simulator. The corresponding URDFs have been downloaded from various repositories and updated/corrected if necessary (adding inertial tags, correcting inertia values, updating collision meshes, etc). All the robots inherit from the main `Robot` class, and can be loaded in the framework using: + +```python +import pyrobolearn as prl + +sim = prl.simulators.BulletSim() +robot = prl.robots.(sim) +``` + +or + +```python +import pyrobolearn as prl + +sim = prl.simulators.BulletSim() +world = prl.worlds.BasicWorld(sim) +robot = world.loadRobot() +``` + +The folder contains different kind of robots including manipulators, legged robots, wheeled robots, and others. It includes: +- [Aibo](https://github.com/dkotfis/aibo_ros) +- [Allegrohand](https://github.com/simlabrobotics/allegro_hand_ros) +- [Ant](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- Atlas: [1](https://github.com/openai/roboschool), [2](https://github.com/erwincoumans/pybullet_robots) +- [Ballbot](https://github.com/CesMak/bb) +- [Baxter](https://github.com/RethinkRobotics/baxter_common) +- BB8: [1](http://www.theconstructsim.com/bb-8-gazebo-model/), [2](https://github.com/eborghi10/BB-8-ROS) +- [Blackbird](https://hackaday.io/project/160882-blackbird-bipedal-robot) +- [Cartpole](https://github.com/bulletphysics/bullet3/blob/master/data/cartpole.urdf) but modified to be able to have multiple links specified at runtime +- Cassie: [1](https://github.com/UMich-BipedLab/Cassie_Model), [2](https://github.com/agilityrobotics/cassie-gazebo-sim), [3](https://github.com/erwincoumans/pybullet_robots) +- [Centauro](https://github.com/ADVRHumanoids/centauro-simulator) +- [Cogimon](https://github.com/ADVRHumanoids/iit-cogimon-ros-pkg) +- [Coman](https://github.com/ADVRHumanoids/iit-coman-ros-pkg) +- [Crab](https://github.com/tuuzdu/crab_project) +- [Cubli](https://github.com/xinsongyan/cubli) +- [Darwin](https://github.com/HumaRobotics/darwin_description) +- [e.Do](https://github.com/Comau/eDO_description) +- [epuck](https://github.com/gctronic/epuck_driver_cpp) +- [F10 racecar](https://github.com/erwincoumans/pybullet_robots/tree/master/data/f10_racecar) +- [Fetch](https://github.com/fetchrobotics/fetch_ros) +- [Franka Emika](https://github.com/frankaemika/franka_ros) +- [Half Cheetah](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- [Hopper](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- [Hubo](https://github.com/robEllenberg/hubo-urdf) +- [Humanoid](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- [Husky](https://github.com/husky/husky) +- [HyQ](https://github.com/iit-DLSLab/hyq-description) +- [HyQ2Max](https://github.com/iit-DLSLab/hyq2max-description) +- ICub: [1](https://github.com/robotology-playground/icub-models), [2](https://github.com/robotology-playground/icub-model-generator). There are currently few problems with this robot. +- [Jaco](https://github.com/JenniferBuehler/jaco-arm-pkgs) +- KR5: [1](https://github.com/a-price/KR5sixxR650WP_description), [2](https://github.com/ros-industrial/kuka_experimental) +- Kuka IIWA: [1](https://github.com/IFL-CAMP/iiwa_stack), [2](https://github.com/bulletphysics/bullet3/tree/master/data/kuka_iiwa) +- Kuka LWR: [1](https://github.com/CentroEPiaggio/kuka-lwr), [2](https://github.com/bulletphysics/bullet3/tree/master/data/kuka_lwr) +- [Laikago](https://github.com/erwincoumans/pybullet_robots) +- [Little Dog](https://github.com/RobotLocomotion/LittleDog) +- [Manipulator2D](https://github.com/domingoesteban/robolearn_robots_ros) +- [Minitaur](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/quadruped) +- [Lincoln MKZ car](https://bitbucket.org/DataspeedInc/dbw_mkz_ros) +- [Morphex](https://gist.github.com/lanius/cb8b5e0ede9ff3b2b2c1bc68b95066fb) +- Nao: [1](https://github.com/ros-naoqi/nao_robot), and [2](https://github.com/ros-naoqi/nao_meshes) +- OpenDog: [1](https://github.com/XRobots/openDog), and [2](https://github.com/wiccopruebas/opendog_project) +- [Pepper](https://github.com/ros-naoqi/pepper_robot) +- [Phantom X](https://github.com/HumaRobotics/phantomx_description) +- [Pleurobot](https://github.com/KM-RoBoTa/pleurobot_ros_pkg) +- [PR2](https://github.com/pr2/pr2_common) +- [Quadcopter](https://github.com/wilselby/ROS_quadrotor_simulator) +- [Rhex](https://github.com/grafoteka/rhex) +- [RRbot](https://github.com/ros-simulation/gazebo_ros_demos) +- Sawyer: [1](https://github.com/RethinkRobotics/sawyer_robot), [2](https://github.com/erwincoumans/pybullet_robots) +- [SEA hexapod](https://github.com/alexansari101/snake_ws) +- [SEA snake]( https://github.com/alexansari101/snake_ws) +- [Soft hand](https://github.com/CentroEPiaggio/pisa-iit-soft-hand) +- [Swimmer](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- [Valkyrie](https://github.com/openhumanoids/val_description) +- [Walker 2D](https://github.com/bulletphysics/bullet3/tree/master/examples/pybullet/gym/pybullet_data/mjcf) +- [Walk-man](https://github.com/ADVRHumanoids/iit-walkman-ros-pkg) +- [Wam](https://github.com/jhu-lcsr/barrett_model) +- [Youbot](https://github.com/youbot): this includes the youbot base without any arms, one kuka arm, 2 kuka arms, and the kuka arm without the wheeled base. + + +Here is a list of robots that I plan to add at one point (some of them require to simulate some fluid dynamics, as done in the `quadcopter` class) but can interest already some people: +- [ ] [ANYmal](https://www.anymal-research.org/): I am currently not sure if I can release the URDF of this robot, and thus I will not do it until it is officially released by ETH. +- [ ] [ROS robots](https://robots.ros.org/): I plan to provide soon the robots listed on this website (I am currently cleaning the URDFs of some of them) +- [ ] [rotors-simulator](https://github.com/ethz-asl/rotors_simulator) +- [ ] [uuv-simulator](https://github.com/uuvsimulator/uuv_simulator) +- [ ] [usv-simulator](https://github.com/OUXT-Polaris/ros_ship_packages) + +Note that currently, we load directly the URDF from the xml/urdf file using the simulator, but later we might first parse it beforehand and allow the user to add, remove, or change few links at runtime. + +TODO: +- [ ] correct still few URDFs (some of them have inertia values that are too high!) +- [ ] finish to implement few methods +- [ ] move all the urdfs outside the pyrobolearn framework: put it in another repo or in a dropbox/google drive. diff --git a/pyrobolearn/robots/__init__.py b/pyrobolearn/robots/__init__.py index fe8d8ae..c242878 100644 --- a/pyrobolearn/robots/__init__.py +++ b/pyrobolearn/robots/__init__.py @@ -31,7 +31,7 @@ from humanoid import Humanoid from aibo import Aibo from minitaur import Minitaur from littledog import LittleDog -# from anymal import ANYmal +# from anymal import ANYmal from hyq import HyQ from hyq2max import HyQ2Max from opendog import OpenDog diff --git a/pyrobolearn/robots/manipulator2d.py b/pyrobolearn/robots/manipulator2d.py new file mode 100644 index 0000000..0f848b1 --- /dev/null +++ b/pyrobolearn/robots/manipulator2d.py @@ -0,0 +1,62 @@ +#!/usr/bin/env python +"""Provide 2d manipulators. +""" + +import os +import numpy as np + +from manipulator import ManipulatorRobot + + +class Manipulator2D(ManipulatorRobot): + r"""2D manipulator robot + + References: + [1] https://github.com/domingoesteban/robolearn_robots_ros + """ + + def __init__(self, + simulator, + init_pos=(0, 0, 0), + init_orient=(0, 0, 0, 1), + useFixedBase=False, + scaling=1., + urdf_path=os.path.dirname(__file__) + '/urdfs/manipulator2d/manipulator2d.urdf'): + # check parameters + if init_pos is None: + init_pos = (0., 0., 0.) + if len(init_pos) == 2: # assume x, y are given + init_pos = tuple(init_pos) + (0.,) + if init_orient is None: + init_orient = (0, 0, 0, 1) + if useFixedBase is None: + useFixedBase = False + + super(Manipulator2D, self).__init__(simulator, urdf_path, init_pos, init_orient, useFixedBase, scaling) + self.name = 'manipulator2d' + + +# Test +if __name__ == "__main__": + from itertools import count + from pyrobolearn.simulators import BulletSim + from pyrobolearn.worlds import BasicWorld + + # Create simulator + sim = BulletSim() + + # create world + world = BasicWorld(sim) + + # create robot + robot = Manipulator2D(sim, init_pos=(0, -0.25, 0)) + robot1 = Manipulator2D(sim, init_pos=(0, 0.25, 0)) + robot.printRobotInfo() + + # Position control using sliders + # robot.addJointSlider() + + # run simulator + for _ in count(): + # robot.updateJointSlider() + world.step(sleep_dt=1./240) diff --git a/pyrobolearn/robots/urdfs/manipulator2d/LICENSE b/pyrobolearn/robots/urdfs/manipulator2d/LICENSE new file mode 100644 index 0000000..91c171d --- /dev/null +++ b/pyrobolearn/robots/urdfs/manipulator2d/LICENSE @@ -0,0 +1,27 @@ +Copyright (c) 2019, Domingo Esteban (domingo.esteban@iit.it) +All rights reserved. + +Redistribution and use in source and binary forms, with or without +modification, are permitted provided that the following conditions are met: +1. Redistributions of source code must retain the above copyright + notice, this list of conditions and the following disclaimer. +2. Redistributions in binary form must reproduce the above copyright + notice, this list of conditions and the following disclaimer in the + documentation and/or other materials provided with the distribution. +3. All advertising materials mentioning features or use of this software + must display the following acknowledgement: + This product includes software developed by the . +4. Neither the name of the nor the + names of its contributors may be used to endorse or promote products + derived from this software without specific prior written permission. + +THIS SOFTWARE IS PROVIDED BY ''AS IS'' AND ANY +EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED +WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE +DISCLAIMED. IN NO EVENT SHALL BE LIABLE FOR ANY +DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES +(INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; +LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND +ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT +(INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS +SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. \ No newline at end of file diff --git a/pyrobolearn/robots/urdfs/manipulator2d/manipulator2d.urdf b/pyrobolearn/robots/urdfs/manipulator2d/manipulator2d.urdf new file mode 100644 index 0000000..24e989c --- /dev/null +++ b/pyrobolearn/robots/urdfs/manipulator2d/manipulator2d.urdf @@ -0,0 +1,480 @@ + + + + + + + + + + + /manipulator2d + gazebo_ros_control/DefaultRobotHWSim + + + + + + Gazebo/Black + + + + Gazebo/White + + + + 0.2 + 0.2 + Gazebo/Blue + + + + Gazebo/White + + + + 0.2 + 0.2 + Gazebo/Blue + + + Gazebo/White + + + + 0.2 + 0.2 + Gazebo/Blue + + + + 0.2 + 0.2 + 0.2 + Gazebo/Red + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + 30.0 + + 1.3962634 + + 800 + 800 + R8G8B8 + + + 0.02 + 300 + + + gaussian + + 0.0 + 0.007 + + + + true + 0.0 + manipulator2d/camera1 + image_raw + camera_info + camera_link_optical + + 0.0 + 0.0 + 0.0 + 0.0 + 0.0 + 0.0 + 0 + 0.0 + 0.0 + 0.0 + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + + transmission_interface/SimpleTransmission + + hardware_interface/EffortJointInterface + + + hardware_interface/EffortJointInterface + 1 + + + + transmission_interface/SimpleTransmission + + hardware_interface/EffortJointInterface + + + hardware_interface/EffortJointInterface + 1 + + + + transmission_interface/SimpleTransmission + + hardware_interface/EffortJointInterface + + + hardware_interface/EffortJointInterface + 1 + + + + diff --git a/pyrobolearn/simulators/README.md b/pyrobolearn/simulators/README.md index 864ce73..8335857 100644 --- a/pyrobolearn/simulators/README.md +++ b/pyrobolearn/simulators/README.md @@ -1,3 +1,13 @@ ## Simulators -This folder contains the APIs to the various simulators. Currently, the main simulator being supported is PyBullet. Work is under progress for Gazebo+ROS, and OpenSIM. \ No newline at end of file +This folder contains the APIs to the various simulators. Currently, the main simulator being supported is PyBullet. Work is under progress for Gazebo+ROS, and OpenSIM. + +```python +import pyrobolearn as prl + +sim = prl.simulators.BulletSim() +``` + +#### What to check next? + +Check the `worlds` folder and the `robots` folder. diff --git a/pyrobolearn/states/README.md b/pyrobolearn/states/README.md index c3f3aa2..b6bd2c9 100644 --- a/pyrobolearn/states/README.md +++ b/pyrobolearn/states/README.md @@ -1,11 +1,32 @@ ## States - The `State` is returned by the environment and given to the policy. The state might include information about the state of one or several objects in the world, including robots. +The `State` is returned by the environment and given to the policy. The state might include information about the state of one or several objects in the world, including robots. It is the main bridge between the robots/objects in the environment and the policy. Specifically, it is given as an input to the policy which knows how to feed the state to the learning model. Usually, the user only has to instantiate a child of this class, and give it to the policy and environment, and that's it. In addition to the policy, the state can be given to a controller, dynamic model, value estimator, reward function, and so on. -To allow the framework to be modular, we favor composition over inheritance [1] leading the state to be decoupled from notions such as the environment, policy, rewards, etc. This class also describes the `state_space` which has initially been defined in `gym.Env` [2]. +To allow the framework to be modular and flexible, we favor [composition over inheritance][https://en.wikipedia.org/wiki/Composition_over_inheritance] leading the state to be decoupled from notions such as the environment, policy, rewards, etc. This class also describes the `state_space` which has initially been defined in [`gym.Env`](https://github.com/openai/gym). -References: - [1] "Wikipedia: Composition over Inheritance", https://en.wikipedia.org/wiki/Composition_over_inheritance - [2] "OpenAI gym": https://gym.openai.com/ and https://github.com/openai/gym \ No newline at end of file + +```python +from itertools import count +import time +import pyrobolearn as prl + + +sim = prl.simulators.BulletSim() +world = prl.worlds.BasicWorld(sim) +robot = prl.robots.loadRobot('wam') # try another robot such as 'coman' or 'littledog' + +state = prl.states.JointPositionState(robot) + prl.states.JointVelocityState(robot) # you can add other states +env = prl.envs.Env(world, state) # the state, world, and possible rewards are defined outside the environment + +for t in count(): + if (t % 240) == 0: + print(state) # print the joint positions and velocities of the specified robot + env.step() # this will ask the state to read or compute the next value, and perform a step in the simulator + time.sleep(1./240) +``` + +#### What to check/look next? + +Check first the `actions` folder, then the `approximators`, `policies`, `rewards`, and `envs` folders. diff --git a/pyrobolearn/tasks/README.md b/pyrobolearn/tasks/README.md new file mode 100644 index 0000000..79117c0 --- /dev/null +++ b/pyrobolearn/tasks/README.md @@ -0,0 +1,13 @@ +## Learning Tasks + +This folder contains learning tasks and how they should be run. These include: +- Reinforcement learning tasks: the task accepts as inputs the environment and the policies. This can then be given to a RL algorithm. +- Imitation learning tasks: the task accepts as inputs the environment, the policies, and the bridges to the interface used to interact with a part of the simulator (robot, object in the world, etc). + +Other learning paradigms include (but are not implemented yet): +- Active learning +- Curriculum learning +- Inverse reinforcement learning +- Transfer learning + +Each task can be evaluated using different metrics. diff --git a/pyrobolearn/tools/README.md b/pyrobolearn/tools/README.md index bc022ff..6bf51db 100644 --- a/pyrobolearn/tools/README.md +++ b/pyrobolearn/tools/README.md @@ -5,4 +5,11 @@ This folder the *interfaces* and the *bridges*. * The I/O *interfaces* get (or set) the data from (to) the hardware, process it, and store it inside the class. * The *bridges* makes the connection between the interface and a component (such as the world or an element in that world such as a robot) in the framework. -The separation between interfaces and bridges allows for better flexibility. For instance, a game controller interface allows us to get data from the hardware, process it, and store it inside the class. The bridge can then map the specific controller events to a robot. Moving a joystick up could mean to move a UAV robot up in the air, or move a wheeled robot forward. \ No newline at end of file +The separation between interfaces and bridges allows for better flexibility. For instance, a game controller interface allows us to get data from the hardware, process it, and store it inside the class. The bridge can then map the specific controller events to a robot. Moving a joystick up could mean to move a UAV robot up in the air, or move a wheeled robot forward. + +While few bridges are already provided in the framework, most of them are let to the user to implement them. + +TODO: +- [ ] clean few interfaces +- [ ] provide other interfaces (FER, google assistant and/or alexa, etc.) +- [ ] provide different bridges diff --git a/pyrobolearn/utils/README.md b/pyrobolearn/utils/README.md new file mode 100644 index 0000000..b9b5e23 --- /dev/null +++ b/pyrobolearn/utils/README.md @@ -0,0 +1,3 @@ +## Utils + +This folder contains several utils methods/classes that are generally useful. diff --git a/pyrobolearn/worlds/README.md b/pyrobolearn/worlds/README.md new file mode 100644 index 0000000..1d24310 --- /dev/null +++ b/pyrobolearn/worlds/README.md @@ -0,0 +1,22 @@ +## Worlds + +In this folder, we provide the `World` class, and the `BasicWorld` class (which loads the floor and sets the gravity). +With this world, you can load floors, visual and collision objects, robots, and others. The world is with the robot (+ actuators/sensors) and the mouse keyboard interface, the only parts in the framework that can interact with the simulator directly. You can load a robot through the world. + +```python +import pyrobolearn as prl + +sim = prl.simulators.BulletSim() +world = prl.worlds.BasicWorld(sim) + +coman = world.loadRobot('Coman', position=) +wam = world.loadRobot(prl.robots.WAM, position=) + +# the following is not advised +littledog = prl.robots.LittleDog(sim, init_pos=) +world.loadRobot(littledog) # such that the world knows about the robot is present. +``` + +#### What to check/look next? + +Check the `robots` folder.