From 8013896982206101a973f8eecc797ac657c43ecc Mon Sep 17 00:00:00 2001 From: cloveranon Date: Tue, 24 Dec 2019 02:49:00 -0500 Subject: [PATCH 1/6] made instructions clearer --- README.md | 32 ++++++++++++++++++++++---------- 1 file changed, 22 insertions(+), 10 deletions(-) diff --git a/README.md b/README.md index 207e419..f7cf175 100644 --- a/README.md +++ b/README.md @@ -38,17 +38,23 @@ To play with GPU, you need an NVIDA GPU with >4 GB of memory (exact minimum requ Windows Installer is hopefully coming someday. I'm not 100% sure how to do it. In the meantime you can manually install pretty easily: -Install python, pytorch (`torch`), and `transformers` (e.g. `pip install torch` (or `pip3`) from the command line, after installing python. Windows users may need to add it to their PATH. Look up how to do these things if you don't know, it's not too hard). Windows users may want to install another module called "colorama", if it is not already installed. If you see `[27m` glyphs, the color codes aren't working, see the color support section. Then: -``` -#git clone "https://github.com/cloveranon/Clover-Edition/" -git clone -cd Clover-Edition -python play.py -``` -(You don't actually need git, just a fast way to clone this repo instead of manually downloading it) -(If that doesn't work try `python3` instead of python. This also assumes git is installed, but you can download a zip file from github and extract it yourself if you don't want to install git.) +Install python, pytorch (`torch`), and `transformers`. Windows users may need to add the location of pip and python to their `PATH` variable manually (look up how to do that, it's not that hard.) Instructions to install pytorch can be found [here](https://pytorch.org/get-started/locally/). You do not need to install the package "torchvision". On Windows, with CUDA 10 and python 3, the command should look like this: -Then you will need to download the model and put it in the models folder rename it to pytorch-gpt2-xl-aid2-v5. The current torrent file and magnet links are here: +``` +pip3 install torch -f https://download.pytorch.org/whl/torch_stable.html +``` + +Then: +``` +pip3 install transformers +``` +That should cover all the dependencies needed to run Clover-Edition. + +Windows users *may* want to install another module called "colorama". Though it may already be installed. If you see `[27m` glyphs, the color codes aren't working See the color support section. + +Then to install just download this repo. Github has a download option somewhere. Or you can use the git command `git clone "https://github.com/cloveranon/Clover-Edition/"` + +Then you will need to download the model. Put it in the models folder. Rename it to `pytorch-gpt2-xl-aid2-v5` until we support arbitrary model names. The current torrent file and magnet links are here: [Torrent File](https://github.com/AccidentallyOnPurpose/pytorch-AIDungeon/blob/f692e39d84/generator/gpt2/models/model_v5_pytorch.torrent?raw=true) [Magnet Link](magnet:?xt=urn:btih:17dcfe3d12849db04a3f64070489e6ff5fc6f63f&dn=model_v5_pytorch&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.cyberia.is%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.moeking.me%3a6969%2fannounce&tr=udp%3a%2f%2f9.rarbg.me%3a2710%2fannounce&tr=udp%3a%2f%2ftracker3.itzmx.com%3a6961%2fannounce) @@ -56,6 +62,12 @@ Then you will need to download the model and put it in the models folder rename magnet:?xt=urn:btih:17dcfe3d12849db04a3f64070489e6ff5fc6f63f&dn=model_v5_pytorch&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.cyberia.is%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.moeking.me%3a6969%2fannounce&tr=udp%3a%2f%2f9.rarbg.me%3a2710%2fannounce&tr=udp%3a%2f%2ftracker3.itzmx.com%3a6961%2fannounce ``` +To play, just enter the folder and run "play.py" with python3. From the command line: +``` +cd Clover-Edition +python3 play.py +``` + Your model folder should look like this: ./models From 8e8896aa38e8d80ea8749941a2c7c769429ee3ff Mon Sep 17 00:00:00 2001 From: cloveranon Date: Tue, 24 Dec 2019 03:23:20 -0500 Subject: [PATCH 2/6] moved some lines in readme --- README.md | 18 ++++++++++-------- 1 file changed, 10 insertions(+), 8 deletions(-) diff --git a/README.md b/README.md index f7cf175..4292ce7 100644 --- a/README.md +++ b/README.md @@ -61,6 +61,16 @@ Then you will need to download the model. Put it in the models folder. Rename it ``` magnet:?xt=urn:btih:17dcfe3d12849db04a3f64070489e6ff5fc6f63f&dn=model_v5_pytorch&tr=udp%3a%2f%2ftracker.opentrackr.org%3a1337%2fannounce&tr=udp%3a%2f%2fopen.stealth.si%3a80%2fannounce&tr=udp%3a%2f%2fp4p.arenabg.com%3a1337%2fannounce&tr=udp%3a%2f%2ftracker.coppersurfer.tk%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.cyberia.is%3a6969%2fannounce&tr=udp%3a%2f%2ftracker.moeking.me%3a6969%2fannounce&tr=udp%3a%2f%2f9.rarbg.me%3a2710%2fannounce&tr=udp%3a%2f%2ftracker3.itzmx.com%3a6961%2fannounce ``` +Your model folder should look like this: +``` + ./models + └── pytorch-gpt2-xl-aid2-v5 + ├── config.json + ├── merges.txt + ├── pytorch_model.bin + └── vocab.json +``` + To play, just enter the folder and run "play.py" with python3. From the command line: ``` @@ -68,14 +78,6 @@ cd Clover-Edition python3 play.py ``` -Your model folder should look like this: - - ./models - └── pytorch-gpt2-xl-aid2-v5 - ├── config.json - ├── merges.txt - ├── pytorch_model.bin - └── vocab.json ##### Color support on Windows (All methods untested. Please report if they do or do not work.): From 3a6f5aa47502d99022b5bf35ff0c1aa950c928a2 Mon Sep 17 00:00:00 2001 From: AccidentallyOnPurpose <4935931+AccidentallyOnPurpose@users.noreply.github.com> Date: Tue, 24 Dec 2019 08:34:17 +0000 Subject: [PATCH 3/6] limit story context to avoid out of mem --- story/story_manager.py | 3 +++ 1 file changed, 3 insertions(+) diff --git a/story/story_manager.py b/story/story_manager.py index 6962c8a..d8e05f3 100644 --- a/story/story_manager.py +++ b/story/story_manager.py @@ -64,6 +64,9 @@ class Story: def add_to_story(self, action, story_block): self.actions.append(action) self.results.append(story_block) + if (len(str(self)) > 3900): # (Fix some mem errors. From RTech, max story of 3900 characters for GTX 2080 ti 11GB + self.actions.pop(1) + self.results.pop(1) def latest_result(self): From 728ce94ad27f9c43a69f707dbbef0bbe4e0f96c8 Mon Sep 17 00:00:00 2001 From: AccidentallyOnPurpose <4935931+AccidentallyOnPurpose@users.noreply.github.com> Date: Tue, 24 Dec 2019 09:03:06 +0000 Subject: [PATCH 4/6] possibly help with out of mem --- play.py | 4 ++++ 1 file changed, 4 insertions(+) diff --git a/play.py b/play.py index ae7fd18..9004bd7 100644 --- a/play.py +++ b/play.py @@ -128,6 +128,10 @@ def play(): colPrint("Go to https://github.com/cloveranon/Clover-Edition/ or email cloveranon@nuke.africa for bug reports, help, and feature requests.", colors['subsubtitle']) while True: + # May be needed to avoid out of mem + gc.collect() + torch.cuda.empty_cache() + if story_manager.story != None: del story_manager.story From e331d0d6214796a92d1aa471da201f2d5207d39e Mon Sep 17 00:00:00 2001 From: cloveranon Date: Tue, 24 Dec 2019 04:29:37 -0500 Subject: [PATCH 5/6] forgot to add pyjarowinkler to intsall instructions. hopefully that covers everything now --- README.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.md b/README.md index 4292ce7..3ccd9be 100644 --- a/README.md +++ b/README.md @@ -44,9 +44,9 @@ Install python, pytorch (`torch`), and `transformers`. Windows users may need to pip3 install torch -f https://download.pytorch.org/whl/torch_stable.html ``` -Then: +Then install `transformers` and `pyjarowinkler`: ``` -pip3 install transformers +pip3 install transformers pyjarowinkler ``` That should cover all the dependencies needed to run Clover-Edition. From ccb4526e50acfa2b75f4a22b4544c315f61a2b8c Mon Sep 17 00:00:00 2001 From: AccidentallyOnPurpose <4935931+AccidentallyOnPurpose@users.noreply.github.com> Date: Tue, 24 Dec 2019 09:50:22 +0000 Subject: [PATCH 6/6] made it download the pytorch model v5 from torrent and rename --- download_model.sh | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/download_model.sh b/download_model.sh index c8b5f29..eca9c6d 100644 --- a/download_model.sh +++ b/download_model.sh @@ -3,13 +3,13 @@ cd "$(dirname "${0}")" BASE_DIR="$(pwd)" BASE_DIR="$(pwd)" -MODELS_DIRECTORY=generator/gpt2/models +MODELS_DIRECTORY=models MODEL_VERSION=model_v5 MODEL_DIRECTORY="${MODELS_DIRECTORY}" -MODEL_NAME=model-550 -MODEL_TORRENT_URL="https://github.com/AIDungeon/AIDungeon/files/3935881/model_v5.torrent.zip" +MODEL_NAME=pytorch-gpt2-xl-aid2-v5 +MODEL_TORRENT_URL="https://raw.githubusercontent.com/AccidentallyOnPurpose/pytorch-AIDungeon/f692e39d84b21d39da9819142165a05a03030892/generator/gpt2/models/model_v5_pytorch.torrent" MODEL_TORRENT_BASENAME="$(basename "${MODEL_TORRENT_URL}")" download_torrent() { @@ -17,7 +17,6 @@ download_torrent() { mkdir -p "${MODEL_DIRECTORY}" cd "${MODEL_DIRECTORY}" wget "${MODEL_TORRENT_URL}" - unzip "${MODEL_TORRENT_BASENAME}" which aria2c > /dev/null if [ $? == 0 ]; then echo -e "\n\n===========================================" @@ -32,8 +31,9 @@ download_torrent() { --seed-time=0 \ --summary-interval=15 \ --disable-ipv6 \ - "${MODEL_TORRENT_BASENAME%.*}" + "${MODEL_TORRENT_BASENAME}" echo "Download Complete!" + mv "${MODEL_TORRENT_BASENAME%.*}" $MODEL_NAME fi }