wassname/erotic-generator
1
0
Fork 0
mirror of https://github.com/wassname/erotic-generator.git synced 2026-09-09 11:22:10 +08:00
Code Issues Packages Projects Releases Wiki Activity

Add poem_generator.ipynb

Browse Source
This commit is contained in:
tfavory
2018-09-10 23:20:25 +08:00
committed by GitHub
parent 9f0b26fdda
commit a7ae42e9fb
1 changed files with 165 additions and 287 deletions
Download Patch File Download Diff File Expand all files Collapse all files
poem_generator.ipynb
+165 -287
View File
@@ -1,24 +1,22 @@
{
"cells": [
{
"cell_type": "code",
"execution_count": 1,
"cell_type": "markdown",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"\u001b[33mThe directory '/home/jovyan/.cache/pip/http' or its parent directory is not owned by the current user and the cache has been disabled. Please check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.\u001b[0m\n",
"\u001b[33mThe directory '/home/jovyan/.cache/pip' or its parent directory is not owned by the current user and caching wheels has been disabled. check the permissions and owner of that directory. If executing pip with sudo, you may want sudo's -H flag.\u001b[0m\n",
"Requirement already satisfied: unidecode in /opt/conda/lib/python3.6/site-packages\n",
"\u001b[33mYou are using pip version 9.0.3, however version 18.0 is available.\n",
"You should consider upgrading via the 'pip install --upgrade pip' command.\u001b[0m\n"
]
}
],
"source": [
"!pip install unidecode"
"# Perth Machine Learning Group Poem Generator\n",
"\n",
"## Introduction\n",
"\n",
"The following code uses GRU to generate poems. It reads through a corpus of poems, and learns sequences of characters, including line breaks and titles.\n",
"\n",
"In short, it observes many sequence of characters, and infers the character that should come next. For instance, it guesses that after 'The cat eat' should come the letter 's'.\n",
"\n",
"Further details will be given with the code.\n",
"\n",
"## The code\n",
"\n",
"### Data exploration"
]
},
{
@@ -27,11 +25,8 @@
"metadata": {},
"outputs": [],
"source": [
"# Import TensorFlow >= 1.9 and enable eager execution\n",
"import tensorflow as tf\n",
"\n",
"# Note: Once you enable eager execution, it cannot be disabled. \n",
"tf.enable_eager_execution()\n",
"import tensorflow as tf # version 1.9 or above\n",
"tf.enable_eager_execution() # Execution of code as it runs in the notebook. Normally, TensorFlow looks up the whole code before execution for efficiency.\n",
"\n",
"import numpy as np\n",
"import re\n",
@@ -46,13 +41,12 @@
"metadata": {},
"outputs": [],
"source": [
"#path_to_file = tf.keras.utils.get_file('shakespeare.txt', 'https://storage.googleapis.com/yashkatariya/shakespeare.txt')\n",
"path_to_file = 'poem_corpus.txt'"
]
},
{
"cell_type": "code",
"execution_count": 4,
"execution_count": 5,
"metadata": {},
"outputs": [
{
@@ -74,49 +68,20 @@
" To judge in peace, or judge in war,\n",
" To judge at night or judge at morn.\n",
" The star that told us of his birth\n",
" Has given us joy and lasting mirth.\n",
"\n",
" The Man that suffered on the tree\n",
" Is risen high above all men;\n",
" Then swell the glad refrain again--\n",
" He died for me, He died for thee:\n",
" Then peace be ever on the earth\n",
" To one and all of human birth.\n",
"\n",
"\n",
"\n",
"\n",
" FALLING OF THE APPLE TREE.\n",
"\n",
"\n",
" The apple tree has fallen, now--\n",
" The axe has laid it low;\n",
" The blossoms sparkled ere it fell,\n",
" But now they wither so.\n",
"\n",
" Its shade we now shall seek in vain--\n",
" The spot we loved so well\n",
" Has\n"
" Has given us joy and lastin\n"
]
}
],
"source": [
"text = unidecode.unidecode(open(path_to_file).read())\n",
"print(text[:1000])"
"print(text[:500])"
]
},
{
"cell_type": "code",
"execution_count": 5,
"cell_type": "markdown",
"metadata": {},
"outputs": [],
"source": [
"# unique contains all the unique characters in the file\n",
"unique = sorted(set(text))\n",
"\n",
"# creating a mapping from unique characters to indices\n",
"char2idx = {u:i for i, u in enumerate(unique)}\n",
"idx2char = {i:u for i, u in enumerate(unique)}"
"### Dataset creation"
]
},
{
@@ -125,169 +90,40 @@
"metadata": {},
"outputs": [],
"source": [
"# setting the maximum length sentence we want for a single input in characters\n",
"max_length = 100\n",
"unique = sorted(set(text)) # unique contains all the unique characters in the corpus\n",
"\n",
"# length of the vocabulary in chars\n",
"vocab_size = len(unique)\n",
"\n",
"# the embedding dimension \n",
"embedding_dim = 256\n",
"\n",
"# number of RNN (here GRU) units\n",
"units = 1024\n",
"\n",
"# batch size \n",
"BATCH_SIZE = 64\n",
"\n",
"# buffer size to shuffle our dataset\n",
"BUFFER_SIZE = 10000"
"char2idx = {u:i for i, u in enumerate(unique)} # maps characters to indexes\n",
"idx2char = {i:u for i, u in enumerate(unique)} # maps indexes to characters"
]
},
{
"cell_type": "code",
"execution_count": 7,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"(15820, 100)\n",
"(15820, 100)\n"
]
}
],
"outputs": [],
"source": [
"input_text = []\n",
"target_text = []\n",
"\n",
"for f in range(0, len(text)-max_length, max_length):\n",
" inps = text[f:f+max_length]\n",
" targ = text[f+1:f+1+max_length]\n",
"\n",
" input_text.append([char2idx[i] for i in inps])\n",
" target_text.append([char2idx[t] for t in targ])\n",
" \n",
"print (np.array(input_text).shape)\n",
"print (np.array(target_text).shape)"
"max_length = 100 # Maximum length sentence we want per input in the network\n",
"vocab_size = len(unique)\n",
"embedding_dim = 256 # number of 'meaningful' features to learn. Ex: ['queen', 'king', 'man', 'woman'] has a least 2 embedding dimension: royalty and gender.\n",
"units = 1024 # In keras: number of output of a sequence. In short it rem\n",
"BATCH_SIZE = 64\n",
"BUFFER_SIZE = 10000"
]
},
{
"cell_type": "code",
"execution_count": 8,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"array([[ 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 27, 32, 42, 33, 43, 44, 37,\n",
" 25, 43, 1, 38, 33, 31, 32, 44, 11, 0, 0, 0, 1, 1, 1, 1, 26,\n",
" 58, 1, 69, 58, 54, 56, 58, 1, 68, 67, 1, 58, 54, 71, 73, 61, 9,\n",
" 1, 60, 68, 68, 57, 1, 76, 62, 65, 65, 1, 73, 68, 1, 66, 58, 67,\n",
" 23, 0, 1, 1, 1, 1, 1, 1, 25, 67, 57, 1, 65, 58, 73, 1, 73,\n",
" 61, 62, 72, 1, 67, 68, 76, 1, 68, 74, 71, 1, 56, 54, 71],\n",
" [68, 65, 1, 55, 58, 22, 0, 1, 1, 1, 1, 1, 1, 33, 59, 1, 68,\n",
" 67, 1, 73, 61, 58, 1, 65, 54, 67, 57, 9, 1, 68, 71, 1, 68, 67,\n",
" 1, 73, 61, 58, 1, 72, 58, 54, 9, 0, 1, 1, 1, 1, 47, 58, 1,\n",
" 72, 73, 62, 65, 65, 1, 76, 62, 65, 65, 1, 72, 62, 67, 60, 1, 73,\n",
" 61, 58, 1, 60, 65, 54, 57, 1, 71, 58, 59, 71, 54, 62, 67, 23, 0,\n",
" 1, 1, 1, 1, 1, 1, 25, 67, 57, 1, 62, 67, 1, 73, 61],\n",
" [58, 1, 56, 65, 68, 72, 62, 67, 60, 1, 65, 62, 60, 61, 73, 1, 68,\n",
" 59, 1, 57, 54, 78, 0, 1, 1, 1, 1, 1, 1, 31, 68, 68, 57, 1,\n",
" 76, 68, 71, 57, 72, 1, 68, 59, 1, 69, 58, 54, 56, 58, 1, 54, 67,\n",
" 57, 1, 56, 61, 58, 58, 71, 1, 76, 62, 65, 65, 1, 72, 54, 78, 11,\n",
" 0, 0, 1, 1, 1, 1, 44, 61, 58, 1, 26, 54, 55, 58, 1, 73, 61,\n",
" 54, 73, 1, 62, 67, 1, 73, 61, 58, 1, 66, 54, 67, 60, 58]])"
]
},
"execution_count": 8,
"metadata": {},
"output_type": "execute_result"
}
],
"outputs": [],
"source": [
"np.array(input_text)[:3]"
]
},
{
"cell_type": "code",
"execution_count": 9,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"1"
]
},
"execution_count": 9,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"np.array(target_text)[0][0]"
]
},
{
"cell_type": "code",
"execution_count": 37,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"'N'"
]
},
"execution_count": 37,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"idx2char[38]"
]
},
{
"cell_type": "code",
"execution_count": 11,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' CHRISTMAS NIGHT.\\n\\n\\n Be peace on earth, good will to men;\\n And let this now our caro'"
]
},
"execution_count": 11,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"''.join(idx2char[target_text[0][i]] for i in range(len(target_text[0])))"
]
},
{
"cell_type": "code",
"execution_count": 12,
"metadata": {},
"outputs": [
{
"data": {
"text/plain": [
"' CHRISTMAS NIGHT.\\n\\n\\n Be peace on earth, good will to men;\\n And let this now our car'"
]
},
"execution_count": 12,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"''.join(idx2char[input_text[0][i]] for i in range(len(input_text[0])))"
"input_text = []\n",
"target_text = []\n",
"\n",
"for f in range(0, len(text) - max_length, max_length):\n",
" inps = text[f : f + max_length]\n",
" targ = text[f + 1 : f + 1 + max_length]\n",
" input_text.append([char2idx[i] for i in inps])\n",
" target_text.append([char2idx[t] for t in targ])"
]
},
{
@@ -310,9 +146,71 @@
"dataset = dataset.apply(tf.contrib.data.batch_and_drop_remainder(BATCH_SIZE))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Explaination\n",
"\n",
"In fact, the algorithm does not learn which characters comes next. It analyzes sequences of characters as inputs (ex: 'abcd'), and predicts sequences as outputs (ex: 'bcde').\n",
"\n",
"Why?\n",
"\n",
"During the training phase, it learns more that just the next character. It updates weights for each characters from the input sequence to the output sequence.\n",
"\n",
"> Consider the sequences 'abcd', 'bcde', 'cdef', 'defg', the letter \"d\" is given different weights that depend on the previous sequences\n",
"\n",
"The use of these updates helps predicting better the next sequences and so on. So it learns the next character but also all the weights \n",
"\n",
"The next chunk of code is optional."
]
},
{
"cell_type": "code",
"execution_count": 23,
"execution_count": 20,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Given the following sequence: \n",
"\n",
"\n",
"ew over the land,\n",
" And the country was wild with glee;\n",
" And she stilled the wave in the stor\n",
"\n",
"\n",
"\n",
"the network learns that a correct continuation is: \n",
"\n",
"w over the land,\n",
" And the country was wild with glee;\n",
" And she stilled the wave in the storm\n"
]
}
],
"source": [
"# example of input:\n",
"print('Given the following sequence: \\n\\n')\n",
"print(''.join(idx2char[input_text[14][i]] for i in range(len(target_text[0]))))\n",
"print('\\n\\n')\n",
"print('the network has to learn that a correct continuation is: \\n')\n",
"# example of output the algorithm has to learn\n",
"print(''.join(idx2char[target_text[14][i]] for i in range(len(input_text[0]))))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Model"
]
},
{
"cell_type": "code",
"execution_count": 14,
"metadata": {},
"outputs": [],
"source": [
@@ -321,77 +219,63 @@
" super(Model, self).__init__()\n",
" self.units = units\n",
" self.batch_sz = batch_size\n",
"\n",
" self.embedding = tf.keras.layers.Embedding(vocab_size, embedding_dim)\n",
"\n",
" if tf.test.is_gpu_available():\n",
" print('GPU available')\n",
" self.gru = tf.keras.layers.CuDNNGRU(self.units, \n",
" return_sequences=True, \n",
" return_state=True, \n",
" recurrent_initializer='glorot_uniform')\n",
" else:\n",
" print('GPU available') \n",
" self.gru = tf.keras.layers.GRU(self.units, \n",
" return_sequences=True, \n",
" return_state=True, \n",
" recurrent_activation='sigmoid', \n",
" recurrent_initializer='glorot_uniform')\n",
"\n",
" self.fc = tf.keras.layers.Dense(vocab_size)\n",
" \n",
" def call(self, x, hidden):\n",
" x = self.embedding(x)\n",
"\n",
" # output shape == (batch_size, max_length, hidden_size) \n",
" # states shape == (batch_size, hidden_size)\n",
"\n",
" # states variable to preserve the state of the model\n",
" # this will be used to pass at every step to the model while training\n",
" output, states = self.gru(x, initial_state=hidden)\n",
"\n",
"\n",
" # reshaping the output so that we can pass it to the Dense layer\n",
" # after reshaping the shape is (batch_size * max_length, hidden_size)\n",
" output = tf.reshape(output, (-1, output.shape[2]))\n",
"\n",
" # The dense layer will output predictions for every time_steps(max_length)\n",
" # output shape after the dense layer == (max_length * batch_size, vocab_size)\n",
" x = self.fc(output)\n",
"\n",
" return x, states"
]
},
{
"cell_type": "code",
"execution_count": 24,
"execution_count": 15,
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"GPU available\n"
]
}
],
"outputs": [],
"source": [
"model = Model(vocab_size, embedding_dim, units, BATCH_SIZE)"
]
},
{
"cell_type": "code",
"execution_count": 25,
"execution_count": 16,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.train.AdamOptimizer()"
]
},
{
"cell_type": "code",
"execution_count": 17,
"metadata": {},
"outputs": [],
"source": [
"optimizer = tf.train.AdamOptimizer()\n",
"\n",
"# using sparse_softmax_cross_entropy so that we don't have to create one-hot vectors\n",
"def loss_function(real, preds):\n",
" return tf.losses.sparse_softmax_cross_entropy(labels=real, logits=preds)"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
"### Training"
]
},
{
"cell_type": "code",
"execution_count": null,
@@ -557,39 +441,41 @@
}
],
"source": [
"# Training step\n",
"n_epochs = 30\n",
"\n",
"EPOCHS = 30\n",
"\n",
"for epoch in range(EPOCHS):\n",
"for epoch in range(n_epochs):\n",
" start = time.time()\n",
" \n",
" # initializing the hidden state at the start of every epoch\n",
" hidden = model.reset_states()\n",
" hidden = model.reset_states() # initializes the hidden state at the start of every epoch\n",
" \n",
" for (batch, (inp, target)) in enumerate(dataset):\n",
" with tf.GradientTape() as tape:\n",
" # feeding the hidden state back into the model\n",
" # This is the interesting step\n",
" predictions, hidden = model(inp, hidden)\n",
" \n",
" # reshaping the target because that's how the \n",
" # loss function expects it\n",
" target = tf.reshape(target, (-1,))\n",
" predictions, hidden = model(inp, hidden) # feeds the hidden state back into the model\n",
" target = tf.reshape(target, (-1, )) # reshapes for the loss function\n",
" loss = loss_function(target, predictions)\n",
" \n",
" grads = tape.gradient(loss, model.variables)\n",
" optimizer.apply_gradients(zip(grads, model.variables), global_step=tf.train.get_or_create_global_step())\n",
"\n",
" if batch % 100 == 0:\n",
" print ('Epoch {} Batch {} Loss {:.4f}'.format(epoch+1,\n",
" batch,\n",
" loss))\n",
" print ('Epoch {} Batch {} Loss {:.4f}'.format(epoch + 1, batch, loss))\n",
" \n",
" print ('Epoch {} Loss {:.4f}'.format(epoch+1, loss))\n",
" print ('Epoch {} Loss {:.4f}'.format(epoch + 1, loss))\n",
" print('Time taken for 1 epoch {} sec\\n'.format(time.time() - start))"
]
},
{
"cell_type": "markdown",
"metadata": {},
"source": [
" ...\n",
" \n",
"The model was trained on Paperspace. 5 epochs are missing due to an average Internet connecton.\n",
"\n",
"Anyway, it is enough to generate some text with the model.\n",
"\n",
"### Text generation"
]
},
{
"cell_type": "code",
"execution_count": 38,
@@ -631,52 +517,44 @@
}
],
"source": [
"num_generate = 1000 # number of characters to generate\n",
"start_string = 'The child' # beginning of the generated text. TODO: try start_string = ' '\n",
"\n",
"input_eval = [char2idx[s] for s in start_string] # converts start_string to numbers the model understands\n",
"input_eval = tf.expand_dims(input_eval, 0) # \n",
"\n",
"# Evaluation step(generating text using the model learned)\n",
"\n",
"# number of characters to generate\n",
"num_generate = 1000\n",
"\n",
"# You can change the start string to experiment\n",
"start_string = 'The child'\n",
"# converting our start string to numbers(vectorizing!) \n",
"input_eval = [char2idx[s] for s in start_string]\n",
"input_eval = tf.expand_dims(input_eval, 0)\n",
"\n",
"# empty string to store our results\n",
"text_generated = ''\n",
"\n",
"# low temperatures results in more predictable text.\n",
"# higher temperatures results in more surprising text\n",
"# experiment to find the best setting\n",
"temperature = 0.97\n",
"temperature = 0.97 # the greater, the closer to an observation in the corpus\n",
"\n",
"# hidden state shape == (batch_size, number of rnn units); here batch size == 1\n",
"hidden = [tf.zeros((1, units))]\n",
"for i in range(num_generate):\n",
" predictions, hidden = model(input_eval, hidden)\n",
" predictions, hidden = model(input_eval, hidden) # predictions holds the probabily for each character to be most adequate continuation\n",
"\n",
" # using a multinomial distribution to predict the word returned by the model\n",
" predictions = predictions / temperature\n",
" predicted_id = tf.multinomial(tf.exp(predictions), num_samples=1)[0][0].numpy()\n",
" predictions = predictions / temperature # alters characters' probabilities to be picked (but keeps the order)\n",
" predicted_id = tf.multinomial(tf.exp(predictions), num_samples=1)[0][0].numpy() # picks the next character for the generated text\n",
" \n",
" # We pass the predicted word as the next input to the model\n",
" # along with the previous hidden state\n",
" input_eval = tf.expand_dims([predicted_id], 0)\n",
" \n",
" text_generated += idx2char[predicted_id]\n",
" text_generated += idx2char[predicted_id] # appends\n",
"\n",
"print (start_string + text_generated)\n",
"\n"
"print (start_string + text_generated)"
]
},
{
"cell_type": "code",
"execution_count": null,
"cell_type": "markdown",
"metadata": {},
"outputs": [],
"source": []
"source": [
"## Conclusion\n",
"\n",
"That's promising:\n",
"* It spells words correctly\n",
"* There is some structure (line breaks).\n",
"* Found a punctation rule\n",
"\n",
"Easy-to-fix issue: indents. The corpus itself is inconsitent for that regard. The fact that the model mimics the indents is in fact a good news.\n",
"\n",
"Harder-to-fix issue: Sentences make little sense. Maybe further training will be enough. Also, playing with hyperparameters will help."
]
}
],
"metadata": {