From daab011f6a629942e3012977dd1f68126a1cecc0 Mon Sep 17 00:00:00 2001 From: wassname Date: Thu, 20 Dec 2018 07:03:18 +0800 Subject: [PATCH] readme etc --- LICENSE.txt | 15 ++++++++++++++ README.md | 54 ++++++++++++++++++++------------------------------ package.json | 10 +++++----- src/index.html | 36 ++++++++++++++++++++++----------- 4 files changed, 65 insertions(+), 50 deletions(-) create mode 100644 LICENSE.txt diff --git a/LICENSE.txt b/LICENSE.txt new file mode 100644 index 0000000..f0b0435 --- /dev/null +++ b/LICENSE.txt @@ -0,0 +1,15 @@ +ISC License + +Copyright (c) 2018, thinkcds.com + +Permission to use, copy, modify, and/or distribute this software for any +purpose with or without fee is hereby granted, provided that the above +copyright notice and this permission notice appear in all copies. + +THE SOFTWARE IS PROVIDED "AS IS" AND THE AUTHOR DISCLAIMS ALL WARRANTIES +WITH REGARD TO THIS SOFTWARE INCLUDING ALL IMPLIED WARRANTIES OF +MERCHANTABILITY AND FITNESS. IN NO EVENT SHALL THE AUTHOR BE LIABLE FOR +ANY SPECIAL, DIRECT, INDIRECT, OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES +WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN +ACTION OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF +OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. diff --git a/README.md b/README.md index b844980..dbe8586 100644 --- a/README.md +++ b/README.md @@ -1,14 +1,23 @@ -# 2d humanoid walker +# 2D Humanoid Walker js + +This project is a reinforcement learning environment for node. You can train a model offline using tensorflow.js then deploy the result to the browser. And example is http://thinkcds.com/2d_walker/. ![Screenshot](docs/img/Selection_146.png) -## Installation +## Installation & Usage -TODO: Describe the installation process +- `git clone https://github.com/wassname/rl_2d_walker.js.git` +- `npm install` +- `npm i -g webpack-dev-server webpack` +- `npm train`: this will train in node + - you ideally need a CUDA capable GPU with 8GB+ of memory. + - will take 8 hours to see results, 44 hours for good results +- `npm run`: this will start a server to view the results. Result will refresh every few hours when a model checkpoint in saved. -## Usage +## Modifying -TODO: Write usage instructions +- The walker: see walker.js +- The agent: see game.js, and ddpg_agent.js ## Contributing @@ -18,38 +27,17 @@ TODO: Write usage instructions 4. Push to the branch: `git push origin my-new-feature` 5. Submit a pull request :D -## TODO - -- [x] make walker into an env, with reset, done etc -- [x] display save loss curves... everything from info -- [x] make diff env's, copy humanoidwalker, humanoid, learning2run etc -- [ ] display - - [x] display/save time for each agent - - [x] globals.step_counter - - [x] x progress - - [ ] smile frown? - - [ ] fix my frameskip - - [x] save checkpoints -- [ ] obstacles later on - - [x] balls - - [ ] Allow user to chuck bouncing balls. Some can remain as obstacles -- [x] load pretrained -- [x] reward hacking - - [x] head high, feet on ground, legs crossing, moving forward, angles near 0 - # Credits +- Made by wassname for thinkcds.com - The walker code is adapted from http://rednuht.org/genetic_walkers/ - DDPG code from metacar -# Notes +# Future work -- Training times: - - Without node: Training: 52732.152ms - - With node: Training: 17211.739ms - - With cuda LoopTime: 12583.449ms - - -- tfjs-node-gpu needs python2 to compile -- crashes during training after a few hours due to mem leak +- [ ] I would like to run the strength, stiffness (in box2d this seems to be max torque), friction, etc to allow a more elegant walking behaviour +- [ ] Clean up the html UI -- is is slipping, what about with more friction? +# License + +ISC diff --git a/package.json b/package.json index 25119bc..1c27630 100644 --- a/package.json +++ b/package.json @@ -1,6 +1,6 @@ { "name": "2d_walker_rl_js", - "version": "0.1.0", + "version": "0.2.0", "description": "A 2d humanoid walker trained to walk with reinforcement learning", "main": "index.js", "dependencies": { @@ -21,16 +21,16 @@ "scripts": { "test": "echo \"Error: no test specified\" && exit 1", "display": "webpack-dev-server -d", - "train": "node --max-old-space-size=4096 scripts/train | tee outputs/train_$(date +\"%Y-%m-%d_%H-%M-%S\").log" + "train": "node --max-old-space-size=8192 scripts/train | tee outputs/train_$(date +\"%Y-%m-%d_%H-%M-%S\").log" }, "repository": { "type": "git", - "url": "git+https://gitlab.com/wassname/2d_humanoidwalker_rl_js.git" + "url": "git+https://github.com/wassname/rl_2d_walker.js.git" }, "author": "", "license": "ISC", "bugs": { - "url": "https://gitlab.com/wassname/2d_humanoidwalker_rl_js/issues" + "url": "https://github.com/wassname/rl_2d_walker.js/issues" }, - "homepage": "http://wassname.org/2d_rl_walker/" + "homepage": "http://thinkcds.com/2d_walker/" } diff --git a/src/index.html b/src/index.html index f411380..00970bb 100644 --- a/src/index.html +++ b/src/index.html @@ -10,25 +10,37 @@

Reinforcement Learning 2D Humanoid Walkers

"Play that funky music, robot"

- +
- Teaching 2d humanoids to walk in your browser using reinforcement learning. You can view the progress at - differen't stages of training with the buttons below. + This project take a teach a 2d ragdoll to walk(ish) using reinforcement learning.

- It's goal it to hold it head upright and move to the right. If you load it's brain from differen't times you can see it's progress. -


- How does it work?: This uses use reinforcement learning to teach them - to walk. This is a branch of machine learning targeted at controlling systems such as system of limbs. Training - is done offline in tensorflow.js. The environment is in box2d for javascript. The aglorithm is DDPG with - prioritized experince replay. + You can view the progress at differen't stages of training with the buttons below. You can also throw balls by clicking the animation.

- The dark outlines are when the agent grips the floor, since I found walking was difficult otherwise. The balls are to provide obstacles. + The agent can move it's limbs in a realistic range of motion, it can feel the position of it's limbs and it's goal it to hold it's head upright and move to the right. + The dark outlines are when the agent grips the floor, since I found walking was slippery otherwise. The balls are to provide obstacles. +
    +
  • Reward: The agent is rewarded for moving to the right, keeping it's head above it's legs, conversing energy, not bending it's limbs too much, and not touching to many limbs to the ground
    • +
  • Actions: The agent can power motors that rotate each limb within a certain range of motion
    • +
  • State: The agent can "see" most things about itself: each limb's relative position, global position, rotation, linear velocity, angular velocity, and orientation. Also each joints angle, speed, and motor speed
    • +


- Credits: The walker code is adapted from rednuht.org/genetic_walkers/, - the DDPG code was adapted from github.com/thibo73800/metacar. + How does it work?: This uses use reinforcement learning to teach the agent to walk. + This is a branch of machine learning targeted at controlling systems over time such as systems of limbs or a self driving car. + + The agent is defined in 2d with a certain strength and range of limb movement. Training is done offline in tensorflow.js. The aglorithm is Deep Deterministic Policy Gradients with prioritized experince replay. The environment is in box2d for javascript and we use webpack to run the same code on the backend and frontend. +

+ +


+ Credits: + +