mirror of
https://github.com/wassname/Clover-Edition.git
synced 2026-10-05 12:20:26 +08:00
restructured
This commit is contained in:
Nick Walton committed
1 parent
fb2ccbf7cf
commit
ca7e5fe6bc
101 files changed
+5
-90822
No files matched your search
@@ -1,3 +0,0 @@
|
||||
seqlen256_v1.ckpt
|
||||
*.txt
|
||||
*.tfrecords
|
||||
@@ -1,85 +0,0 @@
|
||||
# Fine-Tuning the Model on Custom Dataset
|
||||
|
||||
|
||||
This folder contains sample code to fine-tune the model on custom data. It is primarily targeted for GPU usage, but there are pointers throughout showing how to run on TPUs as well.
|
||||
|
||||
Fine-tuning can be used to augment existing control codes or add new control codes. There are 5 steps elaborated upon in the example below:
|
||||
|
||||
1. Patch `keras.py` as in the generation script
|
||||
2. Obtain raw versions of your text files
|
||||
3. Convert this text data into TFRecords; _if you wish to use TPUs, you must transfer these records to GCS._
|
||||
4. Fine-tuning the model on these TFRecords files
|
||||
5. Testing that the generation works.
|
||||
|
||||
## Example of adding a new control code
|
||||
|
||||
Let's begin by adding a new control code `Moby` that is associated with the book [Moby Dick](https://www.gutenberg.org/ebooks/2701)
|
||||
|
||||
If you run `generation.py` with the pretrained models available and try to use this control code, you will find that the model outputs gibberish. Great! It is indeed a fresh new control code. We will run this again after training as a sanity check.
|
||||
|
||||
### Step 1 - Patch your `keras.py`
|
||||
As is required for the generation script, you must patch your `keras.py`. If you patched it before, please roll-back and re-patch with the latest version.
|
||||
|
||||
There are two changes: (1) it defaults to `use_tpu=False` so training/inference takes place on GPUs, (2) the batch size defaults to 4 for GPU training. You might need to go lower depending on your machine.
|
||||
|
||||
You can leave `use_tpu=True` if you wish to train on TPUs and adjust the batch size accordingly.
|
||||
|
||||
### Step 2 - Obtain Your Data
|
||||
|
||||
The book is available publicly; you can simply download it as
|
||||
|
||||
```
|
||||
wget -O moby_dick.txt https://www.gutenberg.org/files/2701/2701-0.txt
|
||||
```
|
||||
|
||||
### Step 3 - Convert Data to TFRecords
|
||||
|
||||
We include the file `make_tf_records.py` to facilitate this.
|
||||
|
||||
Run:
|
||||
|
||||
```
|
||||
python make_tf_records.py --text_file moby_dick.txt --control_code Moby --sequence_len 256
|
||||
```
|
||||
|
||||
It has three arguments: `text_file` which specifies the name of the file to convert, `control_code` which specifies one token (must be in vocabulary) to append to each example, and `sequence_len` which specifies the sequence length to use to create the data. This must match the sequence length of the model being trained.
|
||||
|
||||
|
||||
### Step 4 - Train!
|
||||
|
||||
Simply run `python training.py --model_dir <path_to_model>.ckpt/ --iterations <number_of_iterations>`
|
||||
|
||||
The script picks up all TFRecords in the current folder and fine-tunes the model provided in the `--model_dir` flag.
|
||||
|
||||
If you intend to use TPUs, you must transfer these TFRecords to GCS and edit the location of the data path used by `input_fn` to the GCS bucket.
|
||||
|
||||
To very important gotchas here:
|
||||
|
||||
1. If you have very limited data, the model will very likely overfit and end up memorizing. At the moment, just keep the `--iterations` flag low, preferably equivalent to one epoch or so.
|
||||
|
||||
2. The model is updated and stored in the same directory, if you don't wish to overwrite your model files, please create a backup before you run the training code.
|
||||
|
||||
### Step 5 - Generate!
|
||||
|
||||
We ran `python training.py --model_dir seqlen256_v1.ckpt/ --iterations 250` and try generating with the `Moby` control code.
|
||||
|
||||
Running with the `Moby` control code and a prompt of `I` yields something reasonable in-domain:
|
||||
|
||||
```
|
||||
Moby I <GENERATION_BEGINS> was a little fellow, and he was a
|
||||
great man, what should that matter? And yet it seemed to me that
|
||||
Queequegs words about his father were true. He had been very angry
|
||||
with him, because the old man would not let him go a-whaling...
|
||||
```
|
||||
|
||||
Providing a prompt also works:
|
||||
|
||||
```
|
||||
Moby Then I realized, it wasn't one white whale but three! <GENERATION_BEGINS> And all three
|
||||
were making straight for my boat, which was now some distance away.
|
||||
|
||||
But the three spouts seemed to be coming from different directions, and
|
||||
as they drew nearer and nearer, their tongues began licking up the
|
||||
brine like so many hungry wolves at a carcass...
|
||||
```
|
||||
|
||||
@@ -1 +0,0 @@
|
||||
{"tree_id": "apoc_seed1", "context": "Long ago the bombs fell and the world ended. You are one of the few who is still alive. You are trying to survive by scavenging among the ruins of what is left behind. ", "first_story_block": "You are standing outside an old rundown gas station at the edge of town. There are old broken cars surrounding the gas station and on the road. You are extremely hungry and haven't eaten in a few days. ", "action_results": [{"action": "look in the gas station for food. ", "result": "You search the gas station for food, but find nothing you can eat.", "action_results": [{"action": "search the cars for food.", "result": "You search the cars for food, you search three without success, but in the glove compartment of the fourth you find a box of crackers.", "action_results": [{"action": "eat the crackers.", "result": "They remind you of when you needed to eat a bland diet after your operation. You pat yourself on where you remember the incision was made. The scar healed long ago, but the brain remembers. The crackers weren't very nutritious, but some fuel in the tank is better than none.", "action_results": [{"action": "Continue to look for food in the building up ahead.", "result": "You find a moldy sandwich and some peppered beef jerky in an old lunchbox.", "action_results": [{"action": "Snack on the jerky while searching for more food.", "result": "The jerky is so hard your tooth falls out.", "action_results": [{"action": "look for a mirror to assess the damage", "result": "Blood is pouring from your mouth and the pain is unbearable.", "action_results": [{"action": "Press your tongue into the abscess to stop the bleeding.", "result": "A foul odor rushing into your nostrils and the pain becomes even worse.", "action_results": []}, {"action": "Look for cloth or paper towel to stuff the gaping wound with.", "result": "You find a cloth on the floor. It looks like a used bandage, and is crusted with dried blood.", "action_results": []}]}, {"action": "use pliers to pull the rest of the tooth", "result": "Passes out from the pain. Falls to the floor", "action_results": [{"action": "Check your things for alcohol or antiseptic.", "result": "You find a bottle of alcohol in your bag. You take a swig from the bottle.", "action_results": []}, {"action": "Search the area for medical supplies.", "result": "You find a bottle of Tylenol and struggle to open it.", "action_results": []}]}]}, {"action": "Scrape the mold from the sandwich, taste a bite to see if it is edible.", "result": "The sandwich is stale. You throw it away and hope you don't get sick.", "action_results": [{"action": "Look for more food.", "result": "There is no more food around, but there's two bottles of beer.", "action_results": [{"action": "Drink one beer.", "result": "You drink the beer, but it tastes funny. You hope you don't get sick later.", "action_results": []}, {"action": "Put the beer in your bag and walk towards the next store.", "result": "You grab the beer, leaving the building and heading toward the next store. You are hungry.", "action_results": []}]}, {"action": "Take some medicine you saved.", "result": "It wasn't enought, you are food poisoned.", "action_results": [{"action": "take more medicine", "result": "The medicine does little to improve your upset stomach.", "action_results": []}, {"action": "ride out the food poisoning", "result": "I try to drink some milk.", "action_results": []}]}]}, {"action": "Eat the peppered beef and throw the sandwich to the side", "result": "You discovered the peppered beef had gone bad when you developed cramps and diarrhea just hours after eating it. Fortunately, the food poisoning will pass and you will be all right.", "action_results": [{"action": "Look for water.", "result": "You find a dirty puddle. You know it is likely contaminated but you are so thirsty you bend down and lap it up.", "action_results": [{"action": "It tastes horrible and within a half hour your stomach begins having awful pains.", "result": "You feel nauseous and begin to vomit loudly.", "action_results": []}, {"action": "You change your mind and make yourself throw up the water. You gather the dirty water in a container and search for a way to boil or purify it.", "result": "You see something that looks like a lighter sticking out of a fanny pack caught on a tree, but it looks like you'll need help to get it down.", "action_results": []}, {"action": "Get on your feet and keep walking.", "result": "You walk on. Hopefully, you can come across more water and food.", "action_results": []}, {"action": "Consider the new lows you're hitting, lapping from a dirty puddle.", "result": "You decide it would be better to boil the water before drinking. You fill a container and proceed to start a fire.", "action_results": []}]}, {"action": "Look for more food.", "result": "You find some rotten vegetables, and nothing more.", "action_results": [{"action": "Throw the rotten vegetables against the wall, in angerLine truncated
|
||||
@@ -1 +0,0 @@
|
||||
{"tree_id": "apoc_seed2", "context": "Long ago the bombs fell and the world ended. You are one of the few who is still alive. You are trying to survive by scavenging among the ruins of what is left behind. ", "first_story_block": "You find a car with the keys in the ignition and try to start it. Nothing happens. You keep looking and find another car that looks like it might work. You manage to get the engine started. ", "action_results": [{"action": "drive around looking for any people or stores that might have supplies.", "result": "You drive around the town and see many boarded up buildings, but no people. Eventually you find a store that you think might have supplies.", "action_results": [{"action": "go inside the store and see if there are any supplies.", "result": "It looks like the store has been ransacked and all the shelves are empty. You manage to find a package of granola bars and a flashlight behind the counter.", "action_results": [{"action": "eat a granola bar.", "result": "It tastes just like a granola bar you enjoyed when you were younger. In an instant, you are transported to a school hiking trip. Your mind is awash by the mountain air and the bundles of sticks and leaves that formed the trail. Those days seem so long ago now...", "action_results": [{"action": "Look for other doors inside the store.", "result": "You find the store manager's door, with a set of keys hanging off the lock. The door, though, seems to be blocked from the inside.", "action_results": [{"action": "Push the door to uncover what is blocking it from the inside", "result": "You hear a scream from the inside: \"Go away! I have a gun!\"", "action_results": [{"action": "Continue to push the door.", "result": "You yell: \"So do I!\" and feel the door give a bit more.", "action_results": [{"action": "kick the door", "result": "You try to scape jumping off the window.", "action_results": []}, {"action": "continue to push the door", "result": "The door doesn't give. You look around to find something to ram against the door.", "action_results": []}]}, {"action": "Try to talk down the person behind the door.", "result": "explains that he is a good person, and tries to approach slowly", "action_results": [{"action": "Put your hands up to show you are not armed.", "result": "You put your hands up, but your tattered, duct-taped Remington 870 shotgun is still within reach. Good thing too, as the man reaches behind his back and pulls out a rusty, long blade. You side step, grabbing the shotgun, as the man thrusts out with the knife.", "action_results": []}, {"action": "Take your crossbow out and keep moving forward.", "result": "You arm yourself with the crossbow. You cannot be too cautious.", "action_results": []}]}]}, {"action": "Knock on the door, while being prepared for what is inside.", "result": "You swing the door open and find nothing there. You let out an audible sigh of relief. The door was only jammed.", "action_results": [{"action": "Search the manager's desk.", "result": "You find a note that was written by the last owner. You open up the note.", "action_results": [{"action": "the note is worthless so you throw it away", "result": "You notice that the managers desk has two drawers. One is empty, however the other is locked.", "action_results": []}, {"action": "the note has a cryptic message on it, but you keep it in case someone else can decode it", "result": "You move on from room to room looking for useful supplies.", "action_results": []}]}, {"action": "Look at the coat rack by the door.", "result": "You find a hat, which will help a little when it gets cold at night, but not the warm coat you were hoping for.", "action_results": [{"action": "start looking for more things", "result": "There is no coat here but there is a worn rucksack hung over a chair.", "action_results": []}, {"action": "go out to look for more places", "result": "You find two suitable places that may have clothing to scavenge.", "action_results": []}]}]}, {"action": "Attempt to break down the door with something heavy.", "result": "You look around and find a piece of wood. You take the wood and ram it against the door. Eventually, it starts to give.", "action_results": [{"action": "Keep ramming.", "result": "need to enter that room", "action_results": [{"action": "Hit the hinges and see if you can knock them loose.", "result": "Suddenly the door falls over, and you can now enter the room.", "action_results": []}, {"action": "Take a running start and ram the door as hard as you can.", "result": "The door opens, and you can't believe what you see inside.", "action_results": []}]}, {"action": "Take a break.", "result": "Sitting in the quiet, suddenly you hear a scratching noise coming from the other side of the manager's office door.", "action_results": [{"action": "Listen and try to identify if there is an animal.", "result": "The scratching intensifies. You notice the door begin to shake against the weight of the creature behind it.", "actiLine truncated
|
||||
@@ -1 +0,0 @@
|
||||
{"tree_id": "apoc_seed3", "context": "Long ago the bombs fell and the world ended. You are one of the few who is still alive. You are trying to survive by scavenging among the ruins of what is left behind. ", "first_story_block": "As you leave the store you notice movement out behind your car. A man steps out with a hungry look in his eyes. His clothes are in tatters and he is carrying a crowbar.", "action_results": [{"action": "run at the man and attack him.", "result": "You run forward to attack the man. His crowbar crashes into your chest and you fall to the ground. He kicks you a few times and then grabs your granola bars and runs off.", "action_results": [{"action": "go after him.", "result": "You go after him. As you try and stand up though you realize you're in extreme pain. You think one of your ribs is broken. You manage to get yourself on your feet, but realize catching him is impossible. ", "action_results": [{"action": "look for something to wrap your chest with.", "result": "There's nothing around, so you resort to taking off your pants and wrapping it around your chest. There's a mild draft right now...", "action_results": [{"action": "Check for other injuries.", "result": "There are some superficial cuts on the arms, but nothing else seems to be broken.", "action_results": [{"action": "Search for clothes or blankets for warmth.", "result": "There are boxes on shelves on the wall. You check them for supplies and eventually come across a tote full of blankets. You take one of the blankets out of the tote.", "action_results": [{"action": "Wrap it around yourself.", "result": "You are starting to feel warm again. You can feel your fingers.", "action_results": [{"action": "Count your fingers just for fun.", "result": "You warm up for a bit before collecting more supplies.", "action_results": []}, {"action": "Turn the boxes into shoes.", "result": "You remove the boxes from the shelves, placing two on your feet as makeshift shoes. Your feet begin to warm up.", "action_results": []}]}, {"action": "Stuff it in your bag.", "result": "Your bag is now too full to hold anything else so you remove the other contents so all you are carrying is the blanket. You look around for something to use as a second bag.", "action_results": [{"action": "Make a rucksack out of a dirty blanket.", "result": "The blanket has too many holes, and will not hold anything.", "action_results": []}, {"action": "Create a backpack out of the cardboard box.", "result": "While the cardboard box is uncomfortable, it serves its purpose to carry your supplies.", "action_results": []}]}]}, {"action": "Find water and rinse the exposed limbs to help clean cuts.", "result": "The water is freezing cold, but at least it helps you recover from the shock of being so brutally attacked.", "action_results": [{"action": "Dry off and build a fire.", "result": "The fire comforts you for a brief moment.", "action_results": [{"action": "Watch as the fire comes to life and begins to chase you.", "result": "You decide to go look for food and water after regaining strength.", "action_results": []}, {"action": "Fall asleep by the fire.", "result": "You stay warm throughout the night and wake up to begin the day. Hunting for food and water is on your agenda.", "action_results": []}, {"action": "i go to the store", "result": "I am unable to find anything valuable to scavange.", "action_results": []}, {"action": "go to the store", "result": "You reach the long abandoned store hoping that you might find food or other supplies still left inside. The windows are shattered, probably from previous scavangers.", "action_results": []}]}, {"action": "Dry off and keep moving.", "result": "It is starting to get dark. You must find shelter soon. As you come up over the hill you see a cave. Looks like a good place for shelter.", "action_results": [{"action": "Walk into the cave.", "result": "You walk into the cave and quickly look around to make sure that the cave is safe.", "action_results": []}, {"action": "Keep walking, leave the cave behind.", "result": "You end up sleeping outside unprotected.", "action_results": []}]}]}, {"action": "Look around you", "result": "There's a small river on your left, running along the road. On your right you can see a huge building a couple of miles away.", "action_results": [{"action": "Stop to take a closer look at the river bank.", "result": "There must be animals near the river, looking for water, so you sit quietly near the bank in the hopes of fresh meat.", "action_results": [{"action": "Find something to use as a weapon.", "result": "You look around for a weapon. You pick up a large stick from the ground.", "action_results": []}, {"action": "Refill your water bottle from the river.", "result": "You bring the bottle to your mouth. The water is muddy, but it's refreshing.", "action_results": []}]}, {"action": "Pass by the river and head for the huge building.", "result": "You need water, but the river is probably contaminatLine truncated
|
||||
@@ -1 +0,0 @@
|
||||
{"tree_id": "apoc_seed4", "context": "Long ago the bombs fell and the world ended. You are one of the few who is still alive. You are trying to survive by scavenging among the ruins of what is left behind. ", "first_story_block": "You walk for two hours and take a break. You've left the town you were in and are now in a more rural area. There's a farmhouse to your right and a forest on your left. ", "action_results": [{"action": "go in the farmhouse and look for food.", "result": "You try to open the farmhouse front door, but it's locked. There's a broken window on the right and a shed behind the house.", "action_results": [{"action": "go through the window into the farmhouse. ", "result": "You break the rest of the glass in the window and try to carefully crawl through. You cut yourself on the edges as you're climbing in, but you make it inside. ", "action_results": [{"action": "search the farmhouse for food. ", "result": "You find meat in the fridge that looks like it's gone bad. You also find a can of peas in the bottom of the pantry.", "action_results": [{"action": "Search the farmhouse for tools and utencils.", "result": "You find a broom in the closet and a wooden spoon in the cupboard.", "action_results": [{"action": "Open the van of peas and use the wooden spoon to eat.", "result": "You get the can of peas open and use the wooden spoon to take a bite of them. They taste horrible, but you are so hungry that you hardly notice.", "action_results": [{"action": "Keep eating the peas.", "result": "As you finish the peas, you feel your stomach begin to turn, and know you are in trouble. You quickly look around for medicine.", "action_results": [{"action": "Force yourself to vomit out the peas.", "result": "You turn to the nearest bush and release the contents of your stomach, the pain subsides but the hunger returns.", "action_results": []}, {"action": "Try to find a place to lie down.", "result": "You lie down in a chair in the corner. It's uncomfortable, but you still fall asleep.", "action_results": []}]}, {"action": "Spit the peas onto the ground and stop eating them.", "result": "You begin to feel nauseous and vomit.", "action_results": [{"action": "realize this is from eating after a long time of not eating and force down the peas", "result": "You fight through the nausea and keep the food down.", "action_results": []}, {"action": "throw the can and it bounces off a case of vegetables you hurriedly rush over to.", "result": "Looking over the vegetables you feel they are by far uneatable.", "action_results": []}]}]}, {"action": "Sharpen the broom handle into a weapon for protection and use the broom head as a fire starter.", "result": "You put the broom into your bag and carry it around in case you need to protect yourself.", "action_results": [{"action": "Move to bedroom to find more stuff", "result": "As you enter the bedroom, you lead with the sharpened broom handle. Luckily it is empty, and you quickly pack up a blanket and jacket you find in the closet.", "action_results": [{"action": "there are thermals to wear in the drawers", "result": "It's warm. You are glad that you found the jacket.", "action_results": []}, {"action": "there are arrows to pick up on the ground", "result": "You walk outside. There is a deadly silence in the air.", "action_results": []}, {"action": "Try to fit some shoes in your bag that is quickly filling up.", "result": "You open the bag. find some shoe which is Ok for the time being. You wear it", "action_results": []}, {"action": "Turn around to pick up your broom.", "result": "You then hear a loud crashing sound coming from nearbye.", "action_results": []}]}, {"action": "Check that doors are locked", "result": "You find one door is unlocked, leading you to a dark staircase.", "action_results": [{"action": "go down the stairs", "result": "You walk down the dark staircase, cobwebs choking you as you struggle to see.", "action_results": []}, {"action": "close the door without going through the doorway", "result": "You continue walking through the building. You hear a noise and pull out your broom.", "action_results": []}]}]}, {"action": "take the spoon and the broom", "result": "A man appears from another room in the house.", "action_results": [{"action": "Confront him - maybe he has things you can steal.", "result": "You pull out your gun and ask the man what he is doing there. You notice his bag, and tell him to empty it.", "action_results": [{"action": "Knock the man unconscious.", "result": "You tie the man up to a chair and wait for him to awaken. You ask him questions about supplies and other information that may be valuable.", "action_results": []}, {"action": "Browse through the stuff the man pulled out of the bag.", "result": "You find nothing of particular value, just the essentials to survive in this wasteland. You notice the man seems almost afraid.", "action_results": []}]}, {"action": "Run away.", "result": "You burst through the front door of the Line truncated
|
||||
@@ -1,172 +0,0 @@
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
import sys
|
||||
|
||||
sys.path.append('../')
|
||||
import tensorflow as tf
|
||||
import os
|
||||
import numpy as np
|
||||
|
||||
tf.enable_eager_execution()
|
||||
import transformer
|
||||
import argparse
|
||||
import pdb
|
||||
import re
|
||||
from collections import Counter
|
||||
from tensorflow.python import debug as tf_debug
|
||||
from tensorflow.python.ops import math_ops
|
||||
from tensorflow.python.ops import embedding_ops
|
||||
import fastBPE
|
||||
import platform
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
parser = argparse.ArgumentParser(description='TensorFlow code for generating from CTRL')
|
||||
parser.add_argument('--model_dir', type=str, required=True,
|
||||
help='location of model checkpoint')
|
||||
parser.add_argument('--seed', type=int, default=1337,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
parser.add_argument('--sequence_len', type=int, default=256,
|
||||
help='sequence len of model being fine-tuned (must match also the TFRecords)')
|
||||
parser.add_argument('--iterations', type=int, default=1000,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
|
||||
args = parser.parse_args()
|
||||
tf.random.set_random_seed(args.seed)
|
||||
os.environ['PYTHONHASHSEED'] = str(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('../vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('../vocab',
|
||||
encoding='utf-8').read().split(
|
||||
'\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print('{} unique words'.format(len(vocab)))
|
||||
|
||||
# length of the vocabulary
|
||||
vocab_size = len(vocab)
|
||||
|
||||
# define the numericalization map
|
||||
# idx2word maps the numericalized ID to the word
|
||||
# word2idx maps the word to the numericalized ID
|
||||
word2idx = {u: i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
# sequence length to use for the transformer
|
||||
# must match the model being fine-tuned
|
||||
seq_length = args.sequence_len
|
||||
|
||||
|
||||
def input_fn(params=None):
|
||||
print('READING!', params)
|
||||
dataset = tf.data.Dataset.list_files(tf.io.gfile.glob('./*.tfrecords'), shuffle=True)
|
||||
|
||||
tf_data = tf.data.TFRecordDataset(dataset)
|
||||
myfeatures = {
|
||||
'input': tf.io.FixedLenFeature([256], tf.int64),
|
||||
'output': tf.io.FixedLenFeature([256], tf.int64)
|
||||
}
|
||||
|
||||
def _parse_text_function(example_proto):
|
||||
blah = tf.io.parse_single_example(example_proto, myfeatures)
|
||||
return blah['input'], blah['output']
|
||||
|
||||
train_data = tf_data.map(_parse_text_function).batch(params['batch_size'], drop_remainder=True).repeat().shuffle(
|
||||
10000) # .prefetch(tf.contrib.data.AUTOTUNE)
|
||||
|
||||
return train_data
|
||||
|
||||
|
||||
# the dimension of the transformer
|
||||
embedding_dim = 1280
|
||||
|
||||
|
||||
# Now, we begin defining the model
|
||||
# we defer the transformer definition to transformer.py
|
||||
# here, we only define the tied softmax layer
|
||||
# this layer ties the softmax weights to the input embeddings
|
||||
class TiedEmbeddingSoftmax(tf.keras.layers.Layer):
|
||||
|
||||
def __init__(self, vocab_size=vocab_size, embedding_size=embedding_dim, **kwargs):
|
||||
with tf.device('/cpu:0'):
|
||||
super(TiedEmbeddingSoftmax, self).__init__()
|
||||
self.w = self.add_weight(name='w', shape=(vocab_size, embedding_size),
|
||||
initializer='random_normal',
|
||||
trainable=True)
|
||||
self.b = self.add_weight(name='b', shape=(vocab_size,),
|
||||
initializer='zeros',
|
||||
trainable=True)
|
||||
|
||||
def call(self, inputs, embed=True):
|
||||
with tf.device('/cpu:0'):
|
||||
if embed:
|
||||
dtype = tf.keras.backend.dtype(inputs)
|
||||
if dtype != 'int32' and dtype != 'int64':
|
||||
inputs = math_ops.cast(inputs, 'int32')
|
||||
return embedding_ops.embedding_lookup(self.w, inputs)
|
||||
else:
|
||||
return tf.tensordot(inputs, tf.transpose(self.w), 1) + self.b
|
||||
|
||||
|
||||
# input for the keras model
|
||||
tokens = tf.keras.layers.Input(shape=(seq_length,), dtype='int32')
|
||||
|
||||
# instantiates a tied softmax class
|
||||
tied_embedding_softmax = TiedEmbeddingSoftmax()
|
||||
|
||||
# embedded tokens, before passing it to the transformer
|
||||
embedded = tied_embedding_softmax(tokens, embed=True)
|
||||
|
||||
# the activations after passing it from the transformer
|
||||
# for some odd reason, TPUs don't play well with specifying the arguments of the Encoder() function
|
||||
# so you have to leave them at their defaults
|
||||
transformed = transformer.Encoder()(embedded, training=False)
|
||||
|
||||
# pass the activations from our tiedsoftmax class
|
||||
# this time with embed=False denoting that we are doing the softmax operation
|
||||
# and not a lookup
|
||||
logits = tied_embedding_softmax(transformed, embed=False)
|
||||
|
||||
# finally, define the Keras model with inputs as tokens and outputs as the logits we just computed
|
||||
model = tf.keras.Model(inputs=tokens, outputs=logits)
|
||||
|
||||
|
||||
# the loss function is a simple categorical crossentropy between the logits and the labels
|
||||
def loss(labels, logits):
|
||||
loss = tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
|
||||
mean = tf.math.reduce_mean(loss)
|
||||
loss = tf.Print(loss, [mean])
|
||||
return loss
|
||||
|
||||
|
||||
# the optimizer is not used since this code only supports inference
|
||||
# however, to compile the model, we still define it
|
||||
optimizer = tf.contrib.estimator.clip_gradients_by_norm(
|
||||
tf.train.AdagradOptimizer(learning_rate=3e-3), 0.25)
|
||||
|
||||
# compile the model with the optimizer and loss
|
||||
model.compile(optimizer=optimizer, loss=loss)
|
||||
print(model.summary())
|
||||
|
||||
# IMPORTANT
|
||||
# this is where the saved model is presented to the code
|
||||
# the model directory should have the model checkpoint and
|
||||
# a checkpoint file
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir)
|
||||
|
||||
# this converts the Keras model to a TensorFlow estimator
|
||||
# this step is critical
|
||||
# remember to patch the TF 1.14 file before running the code, else you're going to see errors here
|
||||
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir,
|
||||
session_config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True),
|
||||
tpu_config=tf.contrib.tpu.TPUConfig(iterations_per_loop=100, num_cores_per_replica=1,
|
||||
input_partition_dims=[[1, 1], [1, 1]], per_host_input_for_training=3))
|
||||
tf.logging.set_verbosity(tf.logging.INFO)
|
||||
|
||||
estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
|
||||
|
||||
estimator_model.train(input_fn=input_fn, steps=args.iterations)
|
||||
|
||||
@@ -1,130 +0,0 @@
|
||||
import numpy as np
|
||||
import os
|
||||
import tensorflow as tf
|
||||
import tqdm
|
||||
import pdb
|
||||
import glob
|
||||
import time
|
||||
import sys
|
||||
import re
|
||||
import argparse
|
||||
import fastBPE
|
||||
import platform
|
||||
import json
|
||||
import sys
|
||||
# insert at 1, 0 is the script path (or '' in REPL)
|
||||
sys.path.insert(1, '../../..')
|
||||
from story.utils import *
|
||||
|
||||
def make_samples_helper(context, story_block, action_results, path, tree_id):
|
||||
|
||||
samples = []
|
||||
|
||||
for i, action_result in enumerate(action_results):
|
||||
new_path = path[:]
|
||||
new_path.append(i)
|
||||
if action_result["result"] is not None:
|
||||
sample = [context, story_block, action_result["action"], action_result["result"]]
|
||||
samples.append(sample)
|
||||
if len(action_result["action_results"]) is not 0:
|
||||
sub_result = make_samples_helper(context, action_result["result"], action_result["action_results"], new_path, tree_id)
|
||||
samples += sub_result
|
||||
|
||||
return samples
|
||||
|
||||
|
||||
def make_samples(tree):
|
||||
# Traverse to the bottom levels of each tree
|
||||
first_story_block = tree["first_story_block"]
|
||||
samples = make_samples_helper(tree["context"], first_story_block, tree["action_results"], [], tree["tree_id"])
|
||||
return samples
|
||||
|
||||
|
||||
def build_tokenized_samples(bpe, tree):
|
||||
samples = make_samples(tree)
|
||||
string_samples = []
|
||||
|
||||
for sample in samples:
|
||||
sample = [string.strip() for string in sample]
|
||||
|
||||
sample[2] = sample[2][0].lower() + sample[2][1:]
|
||||
sample[2] = "You " + sample[2]
|
||||
|
||||
new_sample = []
|
||||
|
||||
for item in sample:
|
||||
new_sample.append(second_to_first_person(item))
|
||||
|
||||
string_samples.append(" ".join(new_sample))
|
||||
|
||||
tokenized_samples = [bpe.apply([sample.encode('ascii', errors='ignore') if not use_py3 else sample])[0] for sample in
|
||||
string_samples] # will NOT work for non-English texts
|
||||
tokenized_samples = [re.findall(r'\S+|\n', sample) for sample in tokenized_samples]
|
||||
tokenized_samples = [list(filter(lambda x: x != u'@@', sample)) for sample in tokenized_samples]
|
||||
|
||||
# Fill samples up to seq_len
|
||||
for sample in tokenized_samples:
|
||||
pad_len = seq_length - len(sample)
|
||||
for _ in range(pad_len):
|
||||
sample.append("\n")
|
||||
|
||||
return tokenized_samples
|
||||
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
paths_to_train_files = ["apoc_seed1.json","apoc_seed2.json","apoc_seed3.json","apoc_seed4.json"]
|
||||
seq_length = 256
|
||||
domain = ["Apocalypse"]
|
||||
|
||||
|
||||
# Build sequences from JSON
|
||||
bpe = fastBPE.fastBPE('../codes', '../vocab')
|
||||
tokenized_samples = []
|
||||
for fname in paths_to_train_files:
|
||||
with open(fname, 'r') as fp:
|
||||
tree = json.load(fp)
|
||||
|
||||
tokenized_samples += build_tokenized_samples(bpe, tree)
|
||||
string_samples = []
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('../vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('../vocab', encoding='utf-8').read().split('\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print ('{} unique words'.format(len(vocab)))
|
||||
|
||||
# Creating a mapping from unique characters to indices
|
||||
word2idx = {u:i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
seq_length = seq_length-1
|
||||
|
||||
def numericalize(x):
|
||||
count = 0
|
||||
for i in x:
|
||||
if i not in word2idx:
|
||||
print(i)
|
||||
count += 1
|
||||
return count>1, [word2idx.get(i, word2idx['<unk>']) for i in x]
|
||||
|
||||
tfrecords_fname = 'action_results.tfrecords'
|
||||
|
||||
total = 0
|
||||
skipped = 0
|
||||
with tf.io.TFRecordWriter(tfrecords_fname) as writer:
|
||||
for sample in tokenized_samples:
|
||||
domain_seq = (domain+sample)[:256+1]
|
||||
flag_input, inputs = numericalize(domain_seq[:-1])
|
||||
flag_output, outputs = numericalize(domain_seq[1:])
|
||||
total += 1
|
||||
if flag_input or flag_output:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
if len(inputs)!=seq_length+1 or len(outputs)!=seq_length+1:
|
||||
break
|
||||
example_proto = tf.train.Example(features=tf.train.Features(feature={'input': tf.train.Feature(int64_list=tf.train.Int64List(value=inputs)),
|
||||
'output': tf.train.Feature(int64_list=tf.train.Int64List(value=outputs))}))
|
||||
writer.write(example_proto.SerializeToString())
|
||||
print('Done')
|
||||
print('Skipped', skipped, 'of', total)
|
||||
@@ -1,81 +0,0 @@
|
||||
import numpy as np
|
||||
import os
|
||||
import tensorflow as tf
|
||||
import tqdm
|
||||
import pdb
|
||||
import glob
|
||||
import time
|
||||
import sys
|
||||
import re
|
||||
import argparse
|
||||
import fastBPE
|
||||
import platform
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
parser = argparse.ArgumentParser(description='TensorFlow code for creating TFRecords data')
|
||||
parser.add_argument('--text_file', type=str, required=True,
|
||||
help='location of text file to convert to TFRecords')
|
||||
parser.add_argument('--control_code', type=str, required=True,
|
||||
help='control code to use for this file. must be in the vocabulary, else it will error out.')
|
||||
parser.add_argument('--sequence_len', type=int, required=True,
|
||||
help='sequence length of model being fine-tuned (256 or 512)')
|
||||
|
||||
args = parser.parse_args()
|
||||
|
||||
|
||||
path_to_train_file = fname = args.text_file
|
||||
domain = [args.control_code]
|
||||
|
||||
train_text = open(path_to_train_file, 'rb').read().decode(encoding='utf-8')
|
||||
bpe = fastBPE.fastBPE('../codes', '../vocab')
|
||||
tokenized_train_text = bpe.apply([train_text.encode('ascii', errors='ignore') if not use_py3 else train_text])[0] # will NOT work for non-English texts
|
||||
# if you want to run non-english text, please tokenize separately using ./fast applybpe and then run this script on the .bpe file with utf8 encoding
|
||||
|
||||
tokenized_train_text = re.findall(r'\S+|\n', tokenized_train_text)
|
||||
tokenized_train_text = list(filter(lambda x: x != u'@@', tokenized_train_text))
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('../vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('../vocab', encoding='utf-8').read().split('\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print ('{} unique words'.format(len(vocab)))
|
||||
|
||||
if args.control_code not in vocab:
|
||||
print('Provided control code is not in the vocabulary')
|
||||
print('Please provide a different one; refer to the vocab file for allowable tokens')
|
||||
sys.exit(1)
|
||||
|
||||
# Creating a mapping from unique characters to indices
|
||||
word2idx = {u:i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
seq_length = args.sequence_len-1
|
||||
|
||||
def numericalize(x):
|
||||
count = 0
|
||||
for i in x:
|
||||
if i not in word2idx:
|
||||
print(i)
|
||||
count += 1
|
||||
return count>1, [word2idx.get(i, word2idx['<unk>']) for i in x]
|
||||
|
||||
tfrecords_fname = fname.lower()+'.tfrecords'
|
||||
|
||||
total = 0
|
||||
skipped = 0
|
||||
with tf.io.TFRecordWriter(tfrecords_fname) as writer:
|
||||
for i in tqdm.tqdm(range(0, len(tokenized_train_text), seq_length)):
|
||||
flag_input, inputs = numericalize(domain+tokenized_train_text[i:i+seq_length])
|
||||
flag_output, outputs = numericalize(tokenized_train_text[i:i+seq_length+1])
|
||||
total += 1
|
||||
if flag_input or flag_output:
|
||||
skipped += 1
|
||||
continue
|
||||
|
||||
if len(inputs)!=seq_length+1 or len(outputs)!=seq_length+1:
|
||||
break
|
||||
example_proto = tf.train.Example(features=tf.train.Features(feature={'input': tf.train.Feature(int64_list=tf.train.Int64List(value=inputs)),
|
||||
'output': tf.train.Feature(int64_list=tf.train.Int64List(value=outputs))}))
|
||||
writer.write(example_proto.SerializeToString())
|
||||
print('Done')
|
||||
print('Skipped', skipped, 'of', total)
|
||||
@@ -1,166 +0,0 @@
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
import sys
|
||||
|
||||
sys.path.append('../')
|
||||
import tensorflow as tf
|
||||
import os
|
||||
import numpy as np
|
||||
|
||||
tf.enable_eager_execution()
|
||||
import transformer
|
||||
import argparse
|
||||
import pdb
|
||||
import re
|
||||
from collections import Counter
|
||||
from tensorflow.python import debug as tf_debug
|
||||
from tensorflow.python.ops import math_ops
|
||||
from tensorflow.python.ops import embedding_ops
|
||||
import fastBPE
|
||||
import platform
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
parser = argparse.ArgumentParser(description='TensorFlow code for generating from CTRL')
|
||||
parser.add_argument('--model_dir', type=str, required=True,
|
||||
help='location of model checkpoint')
|
||||
parser.add_argument('--seed', type=int, default=1337,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
parser.add_argument('--sequence_len', type=int, default=256,
|
||||
help='sequence len of model being fine-tuned (must match also the TFRecords)')
|
||||
parser.add_argument('--iterations', type=int, default=1000,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
|
||||
args = parser.parse_args()
|
||||
tf.random.set_random_seed(args.seed)
|
||||
os.environ['PYTHONHASHSEED'] = str(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('../vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('../vocab',
|
||||
encoding='utf-8').read().split(
|
||||
'\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print('{} unique words'.format(len(vocab)))
|
||||
|
||||
# length of the vocabulary
|
||||
vocab_size = len(vocab)
|
||||
|
||||
# define the numericalization map
|
||||
# idx2word maps the numericalized ID to the word
|
||||
# word2idx maps the word to the numericalized ID
|
||||
word2idx = {u: i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
# sequence length to use for the transformer
|
||||
# must match the model being fine-tuned
|
||||
seq_length = args.sequence_len
|
||||
|
||||
|
||||
def input_fn(params=None):
|
||||
print('READING!', params)
|
||||
dataset = tf.data.Dataset.list_files(tf.io.gfile.glob('./*.tfrecords'), shuffle=True)
|
||||
|
||||
tf_data = tf.data.TFRecordDataset(dataset)
|
||||
myfeatures = {
|
||||
'input': tf.io.FixedLenFeature([256], tf.int64),
|
||||
'output': tf.io.FixedLenFeature([256], tf.int64)
|
||||
}
|
||||
|
||||
def _parse_text_function(example_proto):
|
||||
blah = tf.io.parse_single_example(example_proto, myfeatures)
|
||||
return blah['input'], blah['output']
|
||||
|
||||
train_data = tf_data.map(_parse_text_function).batch(params['batch_size'], drop_remainder=True).repeat().shuffle(
|
||||
10000) # .prefetch(tf.contrib.data.AUTOTUNE)
|
||||
|
||||
return train_data
|
||||
|
||||
|
||||
# the dimension of the transformer
|
||||
embedding_dim = 1280
|
||||
|
||||
|
||||
# Now, we begin defining the model
|
||||
# we defer the transformer definition to transformer.py
|
||||
# here, we only define the tied softmax layer
|
||||
# this layer ties the softmax weights to the input embeddings
|
||||
class TiedEmbeddingSoftmax(tf.keras.layers.Layer):
|
||||
|
||||
def __init__(self, vocab_size=vocab_size, embedding_size=embedding_dim, **kwargs):
|
||||
super(TiedEmbeddingSoftmax, self).__init__()
|
||||
self.w = self.add_weight(name='w', shape=(vocab_size, embedding_size),
|
||||
initializer='random_normal',
|
||||
trainable=True)
|
||||
self.b = self.add_weight(name='b', shape=(vocab_size,),
|
||||
initializer='zeros',
|
||||
trainable=True)
|
||||
|
||||
def call(self, inputs, embed=True):
|
||||
if embed:
|
||||
dtype = tf.keras.backend.dtype(inputs)
|
||||
if dtype != 'int32' and dtype != 'int64':
|
||||
inputs = math_ops.cast(inputs, 'int32')
|
||||
return embedding_ops.embedding_lookup(self.w, inputs)
|
||||
else:
|
||||
return tf.tensordot(inputs, tf.transpose(self.w), 1) + self.b
|
||||
|
||||
|
||||
# input for the keras model
|
||||
tokens = tf.keras.layers.Input(shape=(seq_length,), dtype='int32')
|
||||
|
||||
# instantiates a tied softmax class
|
||||
tied_embedding_softmax = TiedEmbeddingSoftmax()
|
||||
|
||||
# embedded tokens, before passing it to the transformer
|
||||
embedded = tied_embedding_softmax(tokens, embed=True)
|
||||
|
||||
# the activations after passing it from the transformer
|
||||
# for some odd reason, TPUs don't play well with specifying the arguments of the Encoder() function
|
||||
# so you have to leave them at their defaults
|
||||
transformed = transformer.Encoder()(embedded, training=False)
|
||||
|
||||
# pass the activations from our tiedsoftmax class
|
||||
# this time with embed=False denoting that we are doing the softmax operation
|
||||
# and not a lookup
|
||||
logits = tied_embedding_softmax(transformed, embed=False)
|
||||
|
||||
# finally, define the Keras model with inputs as tokens and outputs as the logits we just computed
|
||||
model = tf.keras.Model(inputs=tokens, outputs=logits)
|
||||
|
||||
|
||||
# the loss function is a simple categorical crossentropy between the logits and the labels
|
||||
def loss(labels, logits):
|
||||
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
|
||||
|
||||
|
||||
# the optimizer is not used since this code only supports inference
|
||||
# however, to compile the model, we still define it
|
||||
optimizer = tf.contrib.estimator.clip_gradients_by_norm(
|
||||
tf.train.AdagradOptimizer(learning_rate=3e-3), 0.25)
|
||||
|
||||
# compile the model with the optimizer and loss
|
||||
model.compile(optimizer=optimizer, loss=loss)
|
||||
print(model.summary())
|
||||
|
||||
# IMPORTANT
|
||||
# this is where the saved model is presented to the code
|
||||
# the model directory should have the model checkpoint and
|
||||
# a checkpoint file
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir)
|
||||
|
||||
# this converts the Keras model to a TensorFlow estimator
|
||||
# this step is critical
|
||||
# remember to patch the TF 1.14 file before running the code, else you're going to see errors here
|
||||
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir,
|
||||
session_config=tf.ConfigProto(allow_soft_placement=True, log_device_placement=True),
|
||||
tpu_config=tf.contrib.tpu.TPUConfig(iterations_per_loop=100, num_cores_per_replica=1,
|
||||
input_partition_dims=[[1, 1], [1, 1]], per_host_input_for_training=3))
|
||||
tf.logging.set_verbosity(tf.logging.INFO)
|
||||
|
||||
estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
|
||||
|
||||
estimator_model.train(input_fn=input_fn, steps=args.iterations)
|
||||
Reference in new issue
Block a user