datasets page added

DATASETS.md

@@ -0,0 +1,40 @@
+# Datasets and information for training and finetuning AI
+----------------------------
+This page needs a lot of work, please feel free to edit it and submit changes.
+
+## UNIX tools
+----------
+Windows users I'm sorry but your software sucks for things that involve manipulating text files. Fortunately you guys have the windows subsystem for linux which allows you to use all the basic unix tools within windows. I'm just throwing some basic info here for noobs that have no idea what to do. Go to [/g/fglt](https://boards.4channel.org/g/catalog#s=fglt) for help.
+
+### `wget`
+---------------------------
+`wget` is THE greatest tool for scraping websites. Read the man page even though it is long, there are so many useful options (`man wget` noobs.)
+You can scrape an entire website with something like this command: `wget -m website`
+
+### `elinks`
+----------------
+Once you have a bunch of scraped html files, you can convert them to txt files using elinks -dump option. Be sure to read the man page. You will probably want -dump-width 999 or as high as it will go, to disable it's default word wrapping.
+
+### `sed`
+------------
+`sed` is useful for filtering unwanted words or patterns out of files using it's substitution command.  You need to use `info sed` to read it's documentation.
+
+### misc
+------------
+`pdftotext` from poppler utils - does what it says on the tin
+`iconv` to convert weird document encodings to ascii. Especially useful with pdf data that often has converted common character combinations to weird unicode ligatures.
+
+
+# Datasets
+------------------------
+The mormon used a dataset of CYOA texts that can still be found in the data folder of this repo. There are references to other datasets as well. A script used to scrape the /r/writingprompts subreddit, a subreddit where people post short stories. That would be good data if anyone can scrape it properly. Unfortunately reddit's API only shows the last 2,000 posts in any subreddit leaving the vast majority of posts hidden.
+
+There are also references in the code to mechanical turk, a website used to pay thirdworlders to enter data for you for pennies.
+
+Large collection of CYOA smut: https://editthis.info/create_your_own_story/Category:Adult_Stories
+	All in the format of a mediawiki. See https://wiki.installgentoo.com/wiki/Wiki_Backups for info on scraping mediawikis
+	Also see the section on wget 
+
+Project Gutenberg has over 60,000 public domain books that have been manually transcribed into text files. Most of these books are quite old. Virtually all pre-1920s.
+
+Some anon a few days ago claimed to have >18,000 sci fi and fantasy novels in an epub format. I can no longer find his post, would love to have that.

README.md

@@ -32,7 +32,7 @@ To play the game locally, it is recommended that you have an nVidia GPU with 12
 
 Windows Installer is hopefully coming someday. I'm not 100% sure how to do it. In the meantime you can manually install pretty easily:
 
-Install python (version 3.7 or lower), tensorflow (1.14 or possibly 1.15 are known to work), numpy, and regex (e.g. `pip install numpy` from the command line, after installing python. Windows users may need to add it to their PATH. Look up how to do these things if you don't know, it's not too hard). Windows users may want to install another module called "colorama", if it is not already installed. See color support section. Then:
+Install python (version 3.7 or lower), tensorflow (1.14 or possibly 1.15 are known to work), numpy, and regex (e.g. `pip install numpy` (or `pip3`) from the command line, after installing python. Windows users may need to add it to their PATH. Look up how to do these things if you don't know, it's not too hard). Windows users may want to install another module called "colorama", if it is not already installed. See color support section. Then:
 ```
 git clone "https://github.com/cloveranon/Clover-Edition/"
 cd Clover-Edition
@@ -41,15 +41,18 @@ python play.py
 (If that doesn't work try `python3` instead of python. This also assumes git is installed, but you can download a zip file from github and extract it yourself if you don't want to install git.)
 (Tell me if you have a problem installing regex. Any version will work so far as I know. I want to remove it. It is used only twice in the code to do something that can be trivially done with pythons built in regular expressions.)
 
-##### Color support on Windows (all methods untested. Please report if they do or do not work):
+##### Color support on Windows (All methods untested. Please report if they do or do not work.):
 
-* Install a python package called `colorama` and it should work. This is already installed by pip. Which I believe is installed automatically with python. Tell me if color works out of the box on windows now
+* Install a python package called `colorama` and it should work. This may already be installed by pip. Which I believe is installed automatically with python. Tell me if color works out of the box on windows now.
 * Install a windows program called "ansi.sys"
 * Windows 10 users can edit a registry key (look up `Registry Editor`) at `HKEY_CURRENT_USER\Console\VirtualTerminalLevel` to `1` to permanently enable color support
 * user a bat program to enable the `ENABLE_VIRTUAL_TERMINAL_PROCESSING` flag via the `SetConsoleMode` API (not sure what the exact .bat command would be), then run the python script. (If someone figures this out I can put it in the repo and windows users can just run it without doing anything.)
-* use the new "Windows Terminal" which allegedly supports color by default and is in beta. You currently have to install itfrom the windows store until it is officially released
+* use the new "Windows Terminal" which allegedly supports color by default and is in beta. You currently have to install it from the windows store until it is officially released
 
+#### Datasets and retraining the AI
+---------------
 
+I threw together a quick page of some tips [here](DATASETS.md). I plan to throw any links to interesting datasets or guides for training and finetuing the AI there.
 
 #### Community
 ------------------------