Datasets and information for training and finetuning AI

This page needs a lot of work, please feel free to edit it and submit changes.

UNIX tools

Windows users I'm sorry but your software sucks for things that involve manipulating text files. Fortunately you guys have the windows subsystem for linux which allows you to use all the basic unix tools within windows. I'm just throwing some basic info here for noobs that have no idea what to do. Go to /g/fglt for help.

`wget`

wget is THE greatest tool for scraping websites. Read the man page even though it is long, there are so many useful options (man wget noobs.) You can scrape an entire website with something like this command: wget -m website

`elinks`

Once you have a bunch of scraped html files, you can convert them to txt files using elinks -dump option. Be sure to read the man page. You will probably want -dump-width 999 or as high as it will go, to disable it's default word wrapping.

`sed`

sed is useful for filtering unwanted words or patterns out of files using it's substitution command. You need to use info sed to read it's documentation.

misc

pdftotext from poppler utils - does what it says on the tin iconv to convert weird document encodings to ascii. Especially useful with pdf data that often has converted common character combinations to weird unicode ligatures.

Datasets

The mormon used a dataset of CYOA texts that can still be found in the data folder of this repo. There are references to other datasets as well. A script used to scrape the /r/writingprompts subreddit, a subreddit where people post short stories. That would be good data if anyone can scrape it properly. Unfortunately reddit's API only shows the last 2,000 posts in any subreddit leaving the vast majority of posts hidden.

There are also references in the code to mechanical turk, a website used to pay thirdworlders to enter data for you for pennies.

Large collection of CYOA smut: https://editthis.info/create_your_own_story/Category:Adult_Stories All in the format of a mediawiki. See https://wiki.installgentoo.com/wiki/Wiki_Backups for info on scraping mediawikis Also see the section on wget

Project Gutenberg has over 60,000 public domain books that have been manually transcribed into text files. Most of these books are quite old. Virtually all pre-1920s.

Some anon a few days ago claimed to have >18,000 sci fi and fantasy novels in an epub format. I can no longer find his post, would love to have that.

2.6 KiB Raw Blame History