mirror of
https://github.com/wassname/Clover-Edition.git
synced 2026-09-11 11:51:53 +08:00
added model code
This commit is contained in:
Nick
Submodule generator/ctrl/model deleted from 98c8d7cbbe
@@ -0,0 +1,105 @@
|
||||
# Salesforce Open Source Community Code of Conduct
|
||||
|
||||
## About the Code of Conduct
|
||||
|
||||
Equality is a core value at Salesforce. We believe a diverse and inclusive
|
||||
community fosters innovation and creativity, and are committed to building a
|
||||
culture where everyone feels included.
|
||||
|
||||
Salesforce open-source projects are committed to providing a friendly, safe, and
|
||||
welcoming environment for all, regardless of gender identity and expression,
|
||||
sexual orientation, disability, physical appearance, body size, ethnicity, nationality,
|
||||
race, age, religion, level of experience, education, socioeconomic status, or
|
||||
other similar personal characteristics.
|
||||
|
||||
The goal of this code of conduct is to specify a baseline standard of behavior so
|
||||
that people with different social values and communication styles can work
|
||||
together effectively, productively, and respectfully in our open source community.
|
||||
It also establishes a mechanism for reporting issues and resolving conflicts.
|
||||
|
||||
All questions and reports of abusive, harassing, or otherwise unacceptable behavior
|
||||
in a Salesforce open-source project may be reported by contacting the Salesforce
|
||||
Open Source Conduct Committee at ossconduct@salesforce.com.
|
||||
|
||||
## Our Pledge
|
||||
|
||||
In the interest of fostering an open and welcoming environment, we as
|
||||
contributors and maintainers pledge to making participation in our project and
|
||||
our community a harassment-free experience for everyone, regardless of gender
|
||||
identity and expression, sexual orientation, disability, physical appearance,
|
||||
body size, ethnicity, nationality, race, age, religion, level of experience, education,
|
||||
socioeconomic status, or other similar personal characteristics.
|
||||
|
||||
## Our Standards
|
||||
|
||||
Examples of behavior that contributes to creating a positive environment
|
||||
include:
|
||||
|
||||
* Using welcoming and inclusive language
|
||||
* Being respectful of differing viewpoints and experiences
|
||||
* Gracefully accepting constructive criticism
|
||||
* Focusing on what is best for the community
|
||||
* Showing empathy toward other community members
|
||||
|
||||
Examples of unacceptable behavior by participants include:
|
||||
|
||||
* The use of sexualized language or imagery and unwelcome sexual attention or
|
||||
advances
|
||||
* Personal attacks, insulting/derogatory comments, or trolling
|
||||
* Public or private harassment
|
||||
* Publishing, or threatening to publish, others' private information—such as
|
||||
a physical or electronic address—without explicit permission
|
||||
* Other conduct which could reasonably be considered inappropriate in a
|
||||
professional setting
|
||||
* Advocating for or encouraging any of the above behaviors
|
||||
|
||||
## Our Responsibilities
|
||||
|
||||
Project maintainers are responsible for clarifying the standards of acceptable
|
||||
behavior and are expected to take appropriate and fair corrective action in
|
||||
response to any instances of unacceptable behavior.
|
||||
|
||||
Project maintainers have the right and responsibility to remove, edit, or
|
||||
reject comments, commits, code, wiki edits, issues, and other contributions
|
||||
that are not aligned with this Code of Conduct, or to ban temporarily or
|
||||
permanently any contributor for other behaviors that they deem inappropriate,
|
||||
threatening, offensive, or harmful.
|
||||
|
||||
## Scope
|
||||
|
||||
This Code of Conduct applies both within project spaces and in public spaces
|
||||
when an individual is representing the project or its community. Examples of
|
||||
representing a project or community include using an official project email
|
||||
address, posting via an official social media account, or acting as an appointed
|
||||
representative at an online or offline event. Representation of a project may be
|
||||
further defined and clarified by project maintainers.
|
||||
|
||||
## Enforcement
|
||||
|
||||
Instances of abusive, harassing, or otherwise unacceptable behavior may be
|
||||
reported by contacting the Salesforce Open Source Conduct Committee
|
||||
at ossconduct@salesforce.com. All complaints will be reviewed and investigated
|
||||
and will result in a response that is deemed necessary and appropriate to the
|
||||
circumstances. The committee is obligated to maintain confidentiality with
|
||||
regard to the reporter of an incident. Further details of specific enforcement
|
||||
policies may be posted separately.
|
||||
|
||||
Project maintainers who do not follow or enforce the Code of Conduct in good
|
||||
faith may face temporary or permanent repercussions as determined by other
|
||||
members of the project's leadership and the Salesforce Open Source Conduct
|
||||
Committee.
|
||||
|
||||
## Attribution
|
||||
|
||||
This Code of Conduct is adapted from the [Contributor Covenant][contributor-covenant-home],
|
||||
version 1.4, available at https://www.contributor-covenant.org/version/1/4/code-of-conduct.html.
|
||||
It includes adaptions and additions from [Go Community Code of Conduct][golang-coc],
|
||||
[CNCF Code of Conduct][cncf-coc], and [Microsoft Open Source Code of Conduct][microsoft-coc].
|
||||
|
||||
This Code of Conduct is licensed under the [Creative Commons Attribution 3.0 License][cc-by-3-us].
|
||||
|
||||
[contributor-covenant-home]: https://www.contributor-covenant.org (https://www.contributor-covenant.org/)
|
||||
[golang-coc]: https://golang.org/conduct
|
||||
[cncf-coc]: https://github.com/cncf/foundation/blob/master/code-of-conduct.md
|
||||
[microsoft-coc]: https://opensource.microsoft.com/codeofconduct/
|
||||
[cc-by-3-us]: https://creativecommons.org/licenses/by/3.0/us/
|
||||
@@ -0,0 +1,12 @@
|
||||
Copyright (c) 2019, Salesforce.com, Inc.
|
||||
All rights reserved.
|
||||
|
||||
Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
|
||||
|
||||
* Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
|
||||
|
||||
* Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
|
||||
|
||||
* Neither the name of Salesforce.com nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
|
||||
|
||||
THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
|
||||
@@ -0,0 +1,356 @@
|
||||
# CTRL - A Conditional Transformer Language Model for Controllable Generation
|
||||
Authors: [Nitish Shirish Keskar](http://keskarnitish.github.io), [Bryan McCann](https://bmccann.github.io/), [Lav Varshney](http://www.varshney.csl.illinois.edu/), [Caiming Xiong](http://www.stat.ucla.edu/~caiming/), and [Richard Socher](https://www.socher.org/)
|
||||
|
||||
## Introduction
|
||||
|
||||
|
||||
Large-scale language models show promising text generation capabilities, but
|
||||
users cannot easily control this generation process. We release *CTRL*, a 1.6 billion-parameter conditional
|
||||
transformer language model, trained to condition on control codes that specify
|
||||
domain, subdomain, entities, relationships between entities, dates, and task-specific behavior. Control codes were derived from structure that naturally co-occurs with raw text, preserving the advantages of unsupervised learning while providing more explicit control over text generation.
|
||||
|
||||
Paper link: https://arxiv.org/abs/1909.05858
|
||||
|
||||
Blog link: https://blog.einstein.ai/introducing-a-conditional-transformer-language-model-for-controllable-generation/
|
||||
|
||||
The code currently supports two functionalities:
|
||||
1. Generating from a trained model, two models are available for download - one with a sequence length of 256 and another with a sequence length of 512 -- they are trained with word-level vocabularies and through a sliding window approach can generate well beyond their trained sequence lengths.
|
||||
2. Source attribution - given a prompt, prints the perplexity of the prompt conditional on each domain control code (see Section 5 of the paper).
|
||||
|
||||
Please refer to the argument flags for more details regarding the options available for either.
|
||||
|
||||
## Table of Contents
|
||||
|
||||
1. [Citation](#citation)
|
||||
2. [License](#license)
|
||||
3. [Questions for Deliberation](#questions-for-deliberation)
|
||||
4. [Usage](#usage)
|
||||
5. [Sample Generations](#generations)
|
||||
6. [Sample Source Attributions](#source-attributions)
|
||||
7. [FAQs](#faqs)
|
||||
8. [Get Involved](#get-involved)
|
||||
|
||||
|
||||
## Citation
|
||||
```
|
||||
@article{keskarCTRL2019,
|
||||
title={{CTRL - A Conditional Transformer Language Model for Controllable Generation}},
|
||||
author={Keskar, Nitish Shirish and McCann, Bryan and Varshney, Lav and Xiong, Caiming and Socher, Richard},
|
||||
journal={arXiv preprint arXiv:1909.05858},
|
||||
year={2019}
|
||||
}
|
||||
```
|
||||
|
||||
|
||||
|
||||
## License
|
||||
The code is released under the BSD-3 License (see `LICENSE.txt` for details), but we also ask that users respect the following:
|
||||
|
||||
This software should not be used to promote or profit from:
|
||||
|
||||
violence, hate, and division,
|
||||
|
||||
environmental destruction,
|
||||
|
||||
abuse of human rights, or
|
||||
|
||||
the destruction of people's physical and mental health.
|
||||
|
||||
We encourage users of this software to tell us about the applications in which they are putting it to use by emailing ctrl-monitoring@salesforce.com, and to use [appropriate](https://arxiv.org/abs/1810.03993) [documentation](https://www.partnershiponai.org/about-ml/) when developing high-stakes applications of this model.
|
||||
|
||||
## Questions for Deliberation
|
||||
|
||||
We consulted extended members of the AI community in the responsible publication of this model. In particular, a preview of a [Partnership on AI (PAI)](http://partnershiponai.org) project relating to AI research publication norms was considered prior to the release of this work. While this PAI project is as-yet unpublished, it is informed by companies, organizations, and people differently affected by artificial intelligence and presents key considerations to evaluate before publishing potentially high-impact research.
|
||||
|
||||
The questions referenced from the early draft of the PAI project included:
|
||||
|
||||
|
||||
1. How do you envision your research being used in the world? Who will use it? How much expertise is required to use it?
|
||||
2. Who will use it?
|
||||
3. Why would they be motivated to replicate / productionize your work?
|
||||
4. How would a science fiction author turn your research into a dystopian story?
|
||||
5. What is the worst way someone could use your research finding, given no resource constraints?
|
||||
6. What are the historical patterns of misuse or application in this area? How can the research be made more robust against such misuse?
|
||||
7. Which populations or communities will this technology negatively affect, deployed in the scenarios you envision? Will some groups be disproportionately affected?
|
||||
|
||||
|
||||
## Usage
|
||||
|
||||
Here are the steps to get generating:
|
||||
|
||||
1. Install the dependencies
|
||||
|
||||
This code relies on [TensorFlow 1.14](https://www.tensorflow.org/install) and [fastBPE](https://github.com/glample/fastBPE).
|
||||
|
||||
TensorFlow can be installed via `pip install tensorflow[-gpu]==1.14`. fastBPE installation instructions can be found in the GitHub repository linked above. We highly recommend experimenting within a virtualenv or Docker image.
|
||||
|
||||
2. Patch the `/usr/local/lib/python2.7/dist-packages/tensorflow_estimator/python/estimator/keras.py` (or equivalent, if installed elsewhere) by running
|
||||
|
||||
```patch -b <path_to_tensorflow_estimator_package>/python/estimator/keras.py estimator.patch```
|
||||
|
||||
We highly recommend experimenting within a virtualenv or Docker image since the workflow involves patching a TensorFlow file to support some custom functionality. This step is not optional; skipping this step will cause errors (irrespective of device).
|
||||
|
||||
|
||||
3. Get the model files from `gs://sf-ctrl/seqlen256_v1.ckpt/` or `gs://sf-ctrl/seqlen512_v1.ckpt/`.
|
||||
|
||||
The model architecture is identical for both checkpoints. The former is trained with lower training sequence length (256) while the latter is trained with a larger one (512). We plan to update the models (with the appropriate version tags) as we continue to train them longer and on more data. **Our current recommendation is to use the `256_v1` model unless you have a strong reason not to. If you have no preference for domain, `Links` is always a good first choice.**
|
||||
|
||||
[With `gsutil` installed](https://cloud.google.com/storage/docs/gsutil_install), you can simply run `gsutil -m cp -r gs://sf-ctrl/seqlen256_v1.ckpt/ .` for copying the model checkpoint over.
|
||||
|
||||
Without `gsutil`, you can follow the route recommended @ https://github.com/salesforce/ctrl/issues/7#issuecomment-531303214
|
||||
|
||||
4. Run the generation script `generation.py` or the source attribution script `source_attribution.py`.
|
||||
|
||||
The `generation.py` prompts the user to input text and then prints the continuation.
|
||||
The `source_attribution.py` promps the user to input text and then prints a sorted list of domains and the perplexity of the text conditional on each individual domain.
|
||||
|
||||
|
||||
## Generations
|
||||
|
||||
The generations and attributions computed below have been generated using the `256` sequence length model. Comparable results can be obtained from the `512` version of the model as well. We demonstrate only a few of the functionalities, especially the control codes. For a complete list of the control codes, and how to use them, please refer to the paper. Note that `<GENERATION_BEGINS>` is only included for demonstrative purposes and is not actually generated by the model.
|
||||
|
||||
1. Links
|
||||
|
||||
```
|
||||
Links In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
|
||||
|
||||
<GENERATION_BEGINS>
|
||||
|
||||
The scientists were able to track down the unicorn population by analyzing their DNA. They found out that they are descended from an ancient species called the “Tapiri,” which is believed to have been extinct for over 10,000 years.
|
||||
|
||||
According to scientists, this discovery proves that there may be other undiscovered creatures on our planet.
|
||||
|
||||
“We believe these animals could still exist and we hope to find them one day,” said Dr. Carlos Jaramillo, director of the National Geographic Society’s Center for Biodiversity Research at the University of Arizona.
|
||||
|
||||
Scientists also confirmed that the Unicorn Genome Project has already identified several genes associated with the animal’s unique horn structure.
|
||||
|
||||
“The genome project will help us understand how the animal evolved its horns so it can better protect itself against predators like jaguars or pumas.”
|
||||
|
||||
Researchers say that if all goes well, the new genetic information should allow them to create a 3D model of the animal.
|
||||
|
||||
“It would give us a much clearer picture about what makes the creature unique.”- Dr David Sillito, lead researcher ...
|
||||
```
|
||||
|
||||
As a generate note, you don't have to use greedy sampling. You can switch to `topk` or `nucleus` through the appropriate argument flags.
|
||||
|
||||
Running it with `--temperature 0.2` and `--topk 5` yields:
|
||||
|
||||
```
|
||||
Links In a shocking finding, scientist discovered a herd of unicorns living in a remote, previously unexplored valley, in the Andes Mountains. Even more surprising to the researchers was the fact that the unicorns spoke perfect English.
|
||||
|
||||
The scientists were able to track down and capture one of these unicorn herds as it roamed through the valley for several days. They then took DNA samples from each animal and compared them with other animals found throughout the region.
|
||||
|
||||
Advertisement
|
||||
|
||||
What they found is truly astounding.
|
||||
|
||||
According to the scientists, there are approximately 50 different species of unicorn, all of which live on two continents — North America and South America...
|
||||
```
|
||||
|
||||
|
||||
2. Wikipedia
|
||||
|
||||
|
||||
```
|
||||
Wikipedia Salesforce Inc. is <GENERATION_BEGINS> a software company that provides cloud computing and business management software. It was founded in 1999 by Marc Benioff as an online retailer of enterprise software, but it has since expanded into other areas such as e-commerce, customer relationship management, and digital media...
|
||||
```
|
||||
|
||||
|
||||
3. Reviews
|
||||
|
||||
```
|
||||
Reviews Rating: 1.0 <GENERATION_BEGINS>
|
||||
I bought this for my son who is a huge fan of the show. He was so excited to get it and when he opened it, we were all very disappointed. The quality of the product is terrible. It looks like something you would buy at a dollar store.
|
||||
Rating: 1.0
|
||||
This item arrived in poor condition. There are several scratches on the front cover as well as some other minor damage...
|
||||
```
|
||||
|
||||
```
|
||||
Reviews Rating: 4.0 <GENERATION_BEGINS>
|
||||
I bought this for my husband and he loves it. He has a small wrist so it is hard to find watches that fit him well. This one fits perfectly.
|
||||
Rating: 4.0
|
||||
The watch looks great but the band was too big even though i had read other reviews about how large the bands were.
|
||||
Rating: 4.0
|
||||
It works well as an everyday watch. It's not very fancy or anything, just simple with no extra bells & whistles.
|
||||
|
||||
```
|
||||
|
||||
4. Books
|
||||
|
||||
```
|
||||
Books Weary with toil, I haste me to my bed, <GENERATION_BEGINS>
|
||||
And sleep the slumber of a weary man.
|
||||
The morning sun is up, and shines on high;
|
||||
My bed is ready, but no friend appears.
|
||||
A little while ago he came in sight,
|
||||
But now he's gone, as if he'd never come.
|
||||
He left his hat upon the table there,
|
||||
His coat was hanging up behind the door;
|
||||
And, when at last I looked around, there stood
|
||||
No one beside me, nor within call.
|
||||
Oh, what shall I do? What can be done?
|
||||
I'm sure that some strange accident has happened.
|
||||
I'll go out into the street again,
|
||||
To seek for him, or call him back again.
|
||||
But, oh, how lonely it will seem
|
||||
When all are gone who once were near me!
|
||||
```
|
||||
|
||||
5. Reddit - r/running
|
||||
```
|
||||
Running Title: <GENERATION_BEGINS> I just ran my first 5k in under 30 minutes.
|
||||
Text: It was a pretty easy run, but it felt great.
|
||||
|
||||
My goal is to run at least 10 miles per week for the next few months.
|
||||
|
||||
I'm not sure if this qualifies as running or not, but it's definitely something that I've been wanting to do since high school.
|
||||
|
||||
Thanks r/running!
|
||||
|
||||
Edit: Thank you all so much for your kind words and advice.
|
||||
|
||||
Edit 2: Wow, thanks guys!
|
||||
```
|
||||
|
||||
6. Reddit - r/fitness
|
||||
```
|
||||
Fitness Title: <GENERATION_BEGINS> How do you get your body to burn fat when you're not working out?
|
||||
Text: I know that the reason why people lose weight is because they are eating less and burning more calories than their bodies need but how does this work for someone who doesn't have a lot of time or energy to exercise?
|
||||
|
||||
For example, if I'm at home doing nothing all day then I'll be able to eat whatever I want without feeling like I've eaten too much. But what happens after that?
|
||||
|
||||
EDIT: Thanks everyone for the responses!
|
||||
```
|
||||
|
||||
|
||||
7. Reddit - r/frugal
|
||||
|
||||
```
|
||||
Saving Title: <GENERATION_BEGINS> How to get a free credit report
|
||||
Text: I have been trying for months now and it seems like no one is willing to give me one.
|
||||
|
||||
Is there any way that you can just go online and fill out the forms yourself or do they require some sort of ID or something?
|
||||
|
||||
Edit: Thanks everyone for your help!
|
||||
|
||||
Edit 2: Thank you all so much for helping me!
|
||||
```
|
||||
|
||||
8. Reddit - r/personalfinance
|
||||
|
||||
```
|
||||
Finance Title: <GENERATION_BEGINS> I have a question about my credit score.
|
||||
Text: So I'm currently in the process of applying for an auto loan and I've been told that if you don't pay your bills on time then they will not approve you.
|
||||
|
||||
My questions are:
|
||||
|
||||
1. How do I know when to start paying off debt?
|
||||
2. What is the best way to get out of debt without having to file bankruptcy?
|
||||
3. Is it possible to refinance my car loan?
|
||||
4. Should I just wait until after school starts so I can take advantage of lower interest rates?
|
||||
5. If so, how long should I wait?
|
||||
|
||||
Thanks
|
||||
```
|
||||
|
||||
9. Questions
|
||||
|
||||
```
|
||||
Questions Q: What is the capital of Australia? <GENERATION_BEGINS>
|
||||
A: Canberra
|
||||
Q: How many people live in Canberra?
|
||||
A: 650,000
|
||||
```
|
||||
|
||||
10. Translation
|
||||
|
||||
```
|
||||
Translation English : This is a natural language processing model that aims to generate coherent text in a controllable manner. ; French : <GENERATION_BEGINS>
|
||||
Il s'agit d'un modèle de traitement du langage naturel qui vise à générer un texte cohérent et contrôlable.
|
||||
```
|
||||
|
||||
```
|
||||
Translation English : This is a natural language processing model that aims to generate coherent text in a controllable manner. ; German : <GENERATION_BEGINS>
|
||||
Es handelt sich um ein natürliches Textverarbeitungssystem, das auf eine einheitliche und kontrollierbare Erzeugung von Text abzielt.
|
||||
```
|
||||
|
||||
|
||||
|
||||
|
||||
## Source Attributions
|
||||
|
||||
1. `I lost 10 lbs! Feeling great!`
|
||||
|
||||
```
|
||||
PROMPT: I lost 10 lbs! Feeling great!
|
||||
Diet ppl = 28.960714
|
||||
Weight ppl = 29.223865
|
||||
Fitness ppl = 36.162671
|
||||
...
|
||||
```
|
||||
|
||||
2. `My landlord is suing me for unpaid rent`
|
||||
|
||||
```
|
||||
PROMPT: My landlord is suing me for unpaid rent
|
||||
Legal ppl = 21.210965
|
||||
Finance ppl = 24.619064
|
||||
Saving ppl = 27.923208
|
||||
...
|
||||
```
|
||||
|
||||
3. `And then I saw him, the man in the mirror.`
|
||||
|
||||
```
|
||||
PROMPT: And then I saw him, the man in the mirror.
|
||||
Horror ppl = 17.919299
|
||||
Scary ppl = 18.587843
|
||||
Writing ppl = 23.154564
|
||||
...
|
||||
```
|
||||
|
||||
4. `Anarchism is an anti-authoritarian political philosophy that rejects hierarchies deemed unjust and advocates their replacement with self-managed, self-governed societies based on voluntary, cooperative institutions.`
|
||||
|
||||
```
|
||||
PROMPT: Anarchism is an anti-authoritarian political philosophy that rejects hierarchies deemed unjust and advocates their replacement with self-managed, self-governed societies based on voluntary, cooperative institutions.
|
||||
Wikipedia ppl = 34.446701
|
||||
News ppl = 34.484165
|
||||
Links ppl = 35.460126
|
||||
...
|
||||
```
|
||||
5. `I love God`
|
||||
|
||||
```
|
||||
PROMPT: I love God
|
||||
Christianity ppl = 55.653985
|
||||
Atheism ppl = 116.811038
|
||||
Confessions ppl = 133.619834
|
||||
...
|
||||
```
|
||||
|
||||
## FAQs
|
||||
(We hope to update this section frequently).
|
||||
|
||||
1. Will you be releasing the training code and data?
|
||||
|
||||
We plan to release the training code soon. We will not be releasing the training data, but we will release tips and scripts related to data collection.
|
||||
|
||||
2. Is a version of the model available in PyTorch?
|
||||
|
||||
Not at the moment, but if we come across an equivalent implementation, we will update this section.
|
||||
|
||||
3. The code errors out.
|
||||
|
||||
Make sure that you have performed the patch as described above. If the error persists, please create a GitHub issue.
|
||||
|
||||
4. The code generates non-sense irrespective of the prompt.
|
||||
|
||||
Make sure that you have (a) provided the right `--model_dir` and that the folder actually exists and has the checkpoint, (b) provided a valid source code as the first token, and (c) tried generating with a simple prompt such as `Links I` or `Books From`. If the error persists, please create a GitHub issue.
|
||||
|
||||
|
||||
|
||||
|
||||
## Get Involved
|
||||
|
||||
Please create a GitHub issue if you have any questions, suggestions, requests or bug-reports.
|
||||
We welcome PRs!
|
||||
+200000
File diff suppressed because it is too large
Load Diff
@@ -0,0 +1,55 @@
|
||||
0.000296793 Pregnancy
|
||||
0.000127197 Christianity
|
||||
0.003084531 Explain
|
||||
0.000180196 Fitness
|
||||
6.88985E-05 Saving
|
||||
0.000217295 Ask
|
||||
8.47981E-05 Ass
|
||||
0.000143097 Joke
|
||||
0.000196096 Questions
|
||||
0.000127197 Thoughts
|
||||
0.000169596 Retail
|
||||
0.000270294 Feminism
|
||||
0.000111298 Writing
|
||||
0.000402791 Atheism
|
||||
1.05998E-06 Netflix
|
||||
0.000365692 Computing
|
||||
0.000132497 Opinion
|
||||
0.000169596 Alone
|
||||
0.000323293 Funny
|
||||
0.000249094 Gaming
|
||||
0.000402791 Human
|
||||
0.000132497 India
|
||||
2.11995E-08 Joker
|
||||
0.000201395 Diet
|
||||
0.000238495 Legal
|
||||
6.35986E-06 Norman
|
||||
3.60392E-07 Tip
|
||||
0.000302093 Weight
|
||||
0.000132497 Movies
|
||||
0.000111298 Running
|
||||
7.41983E-05 Science
|
||||
0.00135147 Horror
|
||||
0.000291493 Confession
|
||||
0.000190796 Finance
|
||||
0.000413391 Politics
|
||||
7.41983E-05 Scary
|
||||
0.000206695 Support
|
||||
6.35986E-05 Technologies
|
||||
0.000243795 Teenage
|
||||
0.000217295 Event
|
||||
0.000206695 Learned
|
||||
0.000121897 Notion
|
||||
0.0847981 Wikipedia
|
||||
0.095927851 Books
|
||||
0.001176574 Extract
|
||||
0.000127197 Confessions
|
||||
0.000227895 Conspiracy
|
||||
0.365691808 Links
|
||||
0.000423991 Narcissus
|
||||
0.000280894 Relationship
|
||||
0.000922179 Relationships
|
||||
0.153696557 Reviews
|
||||
0.043877717 News
|
||||
0.129847091 Translation
|
||||
0.111297507 multilingual
|
||||
@@ -0,0 +1,24 @@
|
||||
47c47
|
||||
<
|
||||
---
|
||||
> import tensorflow as tf
|
||||
228c228
|
||||
< def _create_keras_model_fn(keras_model, custom_objects=None):
|
||||
---
|
||||
> def _create_keras_model_fn(keras_model, params=None, custom_objects=None):
|
||||
239c239
|
||||
< def model_fn(features, labels, mode):
|
||||
---
|
||||
> def model_fn(features, labels, mode, params=None):
|
||||
448c448
|
||||
< if keras_model._is_graph_network:
|
||||
---
|
||||
> if False:
|
||||
462,464c462,464
|
||||
< estimator = estimator_lib.Estimator(keras_model_fn,
|
||||
< config=config,
|
||||
< warm_start_from=warm_start_path)
|
||||
---
|
||||
> estimator = tf.contrib.tpu.TPUEstimator(keras_model_fn, use_tpu=True, train_batch_size=512, eval_batch_size=32,
|
||||
> config=config,
|
||||
> warm_start_from=warm_start_path)
|
||||
Submodule
+1
Submodule generator/ctrl/model/fastBPE added at 1fd33189c1
@@ -0,0 +1,280 @@
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
import tensorflow as tf
|
||||
import os
|
||||
import numpy as np
|
||||
tf.enable_eager_execution()
|
||||
import transformer
|
||||
import argparse
|
||||
import pdb
|
||||
import sys
|
||||
import re
|
||||
from collections import Counter
|
||||
from tensorflow.python import debug as tf_debug
|
||||
from tensorflow.python.ops import math_ops
|
||||
from tensorflow.python.ops import embedding_ops
|
||||
import fastBPE
|
||||
import platform
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
parser = argparse.ArgumentParser(description='TensorFlow code for generating from CTRL')
|
||||
parser.add_argument('--model_dir', type=str, required=True,
|
||||
help='location of model checkpoint')
|
||||
parser.add_argument('--seed', type=int, default=1337,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
parser.add_argument('--generate_num', type=int, default=256,
|
||||
help='number of tokens to generate')
|
||||
parser.add_argument('--temperature', type=float, default=0,
|
||||
help='temperature for sampling distribution; 0 means greedy')
|
||||
parser.add_argument('--nucleus', type=float, default=0.,
|
||||
help='cumulative probability cutoff for nucleus sampling; 0 means no nucleus sampling')
|
||||
parser.add_argument('--topk', type=int, default=0,
|
||||
help='topk value for sampling from the softmax distribution ; 0 means no topk preferred')
|
||||
parser.add_argument('--penalty', type=float, default=1.2,
|
||||
help='repetition penalty for greedy sampling')
|
||||
|
||||
args = parser.parse_args()
|
||||
tf.random.set_random_seed(args.seed)
|
||||
os.environ['PYTHONHASHSEED'] = str(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('vocab', encoding='utf-8').read().split('\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print ('{} unique words'.format(len(vocab)))
|
||||
|
||||
# length of the vocabulary
|
||||
vocab_size = len(vocab)
|
||||
|
||||
# define the numericalization map
|
||||
# idx2word maps the numericalized ID to the word
|
||||
# word2idx maps the word to the numericalized ID
|
||||
word2idx = {u:i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
|
||||
|
||||
# sequence length to use for the transformer
|
||||
# the model is trained with a seq_length of 512
|
||||
# so, any value <= 512 should work
|
||||
seq_length = min(args.generate_num, 256)
|
||||
|
||||
|
||||
|
||||
|
||||
# the dimension of the transformer
|
||||
embedding_dim = 1280
|
||||
|
||||
|
||||
# Now, we begin defining the model
|
||||
# we defer the transformer definition to transformer.py
|
||||
# here, we only define the tied softmax layer
|
||||
# this layer ties the softmax weights to the input embeddings
|
||||
class TiedEmbeddingSoftmax(tf.keras.layers.Layer):
|
||||
|
||||
def __init__(self, vocab_size=vocab_size, embedding_size=embedding_dim, **kwargs):
|
||||
super(TiedEmbeddingSoftmax, self).__init__()
|
||||
self.w = self.add_weight(name='w', shape=(vocab_size, embedding_size),
|
||||
initializer='random_normal',
|
||||
trainable=True)
|
||||
self.b = self.add_weight(name='b', shape=(vocab_size,),
|
||||
initializer='zeros',
|
||||
trainable=True)
|
||||
|
||||
def call(self, inputs, embed=True):
|
||||
if embed:
|
||||
dtype = tf.keras.backend.dtype(inputs)
|
||||
if dtype != 'int32' and dtype != 'int64':
|
||||
inputs = math_ops.cast(inputs, 'int32')
|
||||
return embedding_ops.embedding_lookup(self.w, inputs)
|
||||
else:
|
||||
return tf.tensordot(inputs, tf.transpose(self.w), 1) + self.b
|
||||
|
||||
# input for the keras model
|
||||
tokens = tf.keras.layers.Input(shape=(seq_length,), dtype='int32')
|
||||
|
||||
# instantiates a tied softmax class
|
||||
tied_embedding_softmax = TiedEmbeddingSoftmax()
|
||||
|
||||
# embedded tokens, before passing it to the transformer
|
||||
embedded = tied_embedding_softmax(tokens, embed=True)
|
||||
|
||||
# the activations after passing it from the transformer
|
||||
# for some odd reason, TPUs don't play well with specifying the arguments of the Encoder() function
|
||||
# so you have to leave them at their defaults
|
||||
transformed = transformer.Encoder()(embedded, training=False)
|
||||
|
||||
|
||||
# pass the activations from our tiedsoftmax class
|
||||
# this time with embed=False denoting that we are doing the softmax operation
|
||||
# and not a lookup
|
||||
logits = tied_embedding_softmax(transformed, embed=False)
|
||||
|
||||
|
||||
# finally, define the Keras model with inputs as tokens and outputs as the logits we just computed
|
||||
model = tf.keras.Model(inputs=tokens, outputs=logits)
|
||||
|
||||
|
||||
# the loss function is a simple categorical crossentropy between the logits and the labels
|
||||
def loss(labels, logits):
|
||||
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
|
||||
|
||||
# the optimizer is not used since this code only supports inference
|
||||
# however, to compile the model, we still define it
|
||||
optimizer = tf.contrib.tpu.CrossShardOptimizer(
|
||||
tf.contrib.estimator.clip_gradients_by_norm(
|
||||
tf.train.AdagradOptimizer(learning_rate=1e-2), 0.25)
|
||||
)
|
||||
|
||||
# compile the model with the optimizer and loss
|
||||
model.compile(optimizer=optimizer, loss=loss)
|
||||
print(model.summary())
|
||||
|
||||
|
||||
# IMPORTANT
|
||||
# this is where the saved model is presented to the code
|
||||
# the model directory should have the model checkpoint and
|
||||
# a checkpoint file
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir)
|
||||
|
||||
|
||||
# this converts the Keras model to a TensorFlow estimator
|
||||
# this step is critical
|
||||
# remember to patch the TF 1.14 file before running the code, else you're going to see errors here
|
||||
estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
|
||||
|
||||
# we now create a serving function from this estimator
|
||||
# this enables us to load the model once and easily query it multiple times
|
||||
def serving_input_fn():
|
||||
inputs = {'input_1': tf.placeholder(tf.int32, [1,seq_length])}
|
||||
return tf.estimator.export.ServingInputReceiver(inputs, inputs)
|
||||
predict_fn = tf.contrib.predictor.from_estimator(estimator_model, serving_input_fn)
|
||||
|
||||
# almost there, we now take the user prompt and tokenize with BPE
|
||||
# load BPE codes
|
||||
bpe = fastBPE.fastBPE('codes', 'vocab')
|
||||
|
||||
temperature = args.temperature
|
||||
nucleusprob = args.nucleus
|
||||
penalty = args.penalty
|
||||
topk = args.topk
|
||||
|
||||
while True:
|
||||
prompt = raw_input('ENTER PROMPT: ') if not use_py3 else input('ENTER PROMPT: ')
|
||||
|
||||
# tokenize provided prompt
|
||||
split_prompt = bpe.apply([prompt])[0].split()
|
||||
text = [word2idx[i] for i in split_prompt]
|
||||
|
||||
# pad with 0s and create a mini-batch of 2 (arbitrary, for ease of code)
|
||||
padded_text = text + [0] * (args.generate_num - len(text))
|
||||
tokens_generated = np.tile(padded_text, (1,1))
|
||||
try:
|
||||
for token in range(len(text)-1, args.generate_num-1):
|
||||
# get the logits from the prediction function
|
||||
# the logic here is a bit convoluted because we are allowing generation past 512 tokens
|
||||
# this is done by sliding the window over (past 512 tokens) and continuing prediction
|
||||
# I'm sure this can be simplified (TODO)
|
||||
if token <= seq_length:
|
||||
prompt_logits = predict_fn({'input_1':tokens_generated[:, :seq_length]})['tied_embedding_softmax'].squeeze() / (temperature if temperature>0 else 1.)
|
||||
_token = token if token < seq_length else -1
|
||||
else:
|
||||
_token = -1
|
||||
end = token + 1
|
||||
start = token - seq_length + 2
|
||||
prompt_logits = predict_fn({'input_1':np.hstack((tokens_generated[:,0:1], tokens_generated[:,start:end]))})['tied_embedding_softmax'].squeeze() / (temperature if temperature>0 else 1.)
|
||||
|
||||
|
||||
# if penalty (for repetition) is non-zero,
|
||||
# discount the logits from already generated tokens
|
||||
if penalty>0:
|
||||
penalized_so_far = set()
|
||||
for _ in range(token+1):
|
||||
generated_token = tokens_generated[0][_]
|
||||
# don't penalize newlines
|
||||
# you could also choose not to penalize frequent words
|
||||
# (which incidentally are sorted in the vocab file)
|
||||
# but I don't do that
|
||||
# if it prints too many new lines instead of continuing generating text,
|
||||
# you might want to comment this out
|
||||
if idx2word[generated_token] == '\n':
|
||||
continue
|
||||
if generated_token in penalized_so_far:
|
||||
continue
|
||||
penalized_so_far.add(generated_token)
|
||||
prompt_logits[_token][generated_token] /= penalty
|
||||
|
||||
# disallow some tokens
|
||||
prompt_logits[_token][word2idx['<unk>']] = -1e8
|
||||
|
||||
# sometimes, when generating from reddit,
|
||||
# it tries to generate the Score (reddit Karma) immediately after generating the Title:
|
||||
# to disallow this, we can just prevent it from generating Score
|
||||
prompt_logits[_token][word2idx['Sco@@']] = -1e8
|
||||
|
||||
|
||||
# compute probabilities from logits
|
||||
prompt_probs = np.exp(prompt_logits[_token])
|
||||
prompt_probs = prompt_probs / sum(prompt_probs)
|
||||
pruned_list = np.argsort(prompt_probs)[::-1]
|
||||
# if you are using nucleus prob, then compute the nucleus probability size
|
||||
if nucleusprob > 0.:
|
||||
minimum_topk = 1
|
||||
nucleus = max(np.where(np.cumsum(np.sort(prompt_probs)[::-1])>nucleusprob)[0][0], minimum_topk)
|
||||
elif topk > 0:
|
||||
# we are over-loading notation here
|
||||
# if you choose to specify a topk instead of a nucleus,
|
||||
# we will hardcode the nucleus to be just that
|
||||
nucleus = topk
|
||||
else:
|
||||
# if you specify neither nucleus or topk,
|
||||
# then we will use the whole list
|
||||
nucleus = len(pruned_list)
|
||||
|
||||
# if you want to disallow more complex tokens, you can do so here
|
||||
# for instance, if you want to disallow anything with the phrase `http`,
|
||||
# you can delete theme from the pruned_list
|
||||
# you can comment this out, I'm keeping it in for demonstration purpose
|
||||
tokens_to_disallow = []
|
||||
for _ in range(len(pruned_list)):
|
||||
if 'http' in idx2word[pruned_list[_]]:
|
||||
tokens_to_disallow.append(_)
|
||||
pruned_list = np.delete(pruned_list, tokens_to_disallow)
|
||||
|
||||
# if temperature is 0
|
||||
# just pick the first (most probable) token
|
||||
if temperature==0:
|
||||
idx = pruned_list[0]
|
||||
else:
|
||||
# else,
|
||||
# sample from the pruned_list with the logits
|
||||
chosen_idx = int(tf.random.categorical(np.expand_dims(prompt_logits[0][_token][pruned_list],0), num_samples=1).numpy())
|
||||
idx = pruned_list[chosen_idx]
|
||||
|
||||
|
||||
# if you want to do some debugging,
|
||||
# like which one was chosen,
|
||||
# what the top25 were,
|
||||
# here is your opportunity.
|
||||
#print('chosen:', idx2word[idx])
|
||||
#print('top25 alternatives:', pruned_list[:25])
|
||||
|
||||
# assign the token for generation
|
||||
tokens_generated[0][token+1] = idx
|
||||
|
||||
# clear screen if you want to
|
||||
# os.system("clear")
|
||||
|
||||
|
||||
tokens_generated_so_far = ' '.join([idx2word[c] for c in tokens_generated[0].squeeze()[:token+2]])
|
||||
tokens_generated_so_far = re.sub('(@@ )', '', string=tokens_generated_so_far)
|
||||
tokens_generated_so_far = re.sub('(@@ ?$)', '', string=tokens_generated_so_far)
|
||||
print(tokens_generated_so_far)
|
||||
print()
|
||||
|
||||
|
||||
except KeyboardInterrupt: #Exception as e:
|
||||
print('Continuing')
|
||||
|
||||
@@ -0,0 +1,192 @@
|
||||
from __future__ import division
|
||||
from __future__ import print_function
|
||||
import tensorflow as tf
|
||||
import os
|
||||
import numpy as np
|
||||
tf.enable_eager_execution()
|
||||
import transformer
|
||||
import argparse
|
||||
import pdb
|
||||
import sys
|
||||
import re
|
||||
from collections import Counter
|
||||
from tensorflow.python import debug as tf_debug
|
||||
from tensorflow.python.ops import math_ops
|
||||
from tensorflow.python.ops import embedding_ops
|
||||
import fastBPE
|
||||
import platform
|
||||
|
||||
use_py3 = platform.python_version()[0] == '3'
|
||||
|
||||
parser = argparse.ArgumentParser(description='TensorFlow code for generating from CTRL')
|
||||
parser.add_argument('--model_dir', type=str, required=True,
|
||||
help='location of model checkpoint')
|
||||
parser.add_argument('--seed', type=int, default=1337,
|
||||
help='random seed for TensorFlow, numpy and PythonHash')
|
||||
|
||||
args = parser.parse_args()
|
||||
tf.random.set_random_seed(args.seed)
|
||||
os.environ['PYTHONHASHSEED'] = str(args.seed)
|
||||
np.random.seed(args.seed)
|
||||
|
||||
# load the vocabulary from file
|
||||
vocab = open('vocab').read().decode(encoding='utf-8').split('\n') if not use_py3 else open('vocab', encoding='utf-8').read().split('\n')
|
||||
vocab = list(map(lambda x: x.split(' ')[0], vocab)) + ['<unk>'] + ['\n']
|
||||
print ('{} unique words'.format(len(vocab)))
|
||||
|
||||
# length of the vocabulary
|
||||
vocab_size = len(vocab)
|
||||
|
||||
# define the numericalization map
|
||||
# idx2word maps the numericalized ID to the word
|
||||
# word2idx maps the word to the numericalized ID
|
||||
word2idx = {u:i for i, u in enumerate(vocab)}
|
||||
idx2word = np.array(vocab)
|
||||
|
||||
|
||||
|
||||
# sequence length to use for the transformer
|
||||
# the model is trained with a seq_length of 512
|
||||
# so, any value <= 512 should work
|
||||
seq_length = 256
|
||||
|
||||
|
||||
|
||||
|
||||
# the dimension of the transformer
|
||||
embedding_dim = 1280
|
||||
|
||||
|
||||
# Now, we begin defining the model
|
||||
# we defer the transformer definition to transformer.py
|
||||
# here, we only define the tied softmax layer
|
||||
# this layer ties the softmax weights to the input embeddings
|
||||
class TiedEmbeddingSoftmax(tf.keras.layers.Layer):
|
||||
|
||||
def __init__(self, vocab_size=vocab_size, embedding_size=embedding_dim, **kwargs):
|
||||
super(TiedEmbeddingSoftmax, self).__init__()
|
||||
self.w = self.add_weight(name='w', shape=(vocab_size, embedding_size),
|
||||
initializer='random_normal',
|
||||
trainable=True)
|
||||
self.b = self.add_weight(name='b', shape=(vocab_size,),
|
||||
initializer='zeros',
|
||||
trainable=True)
|
||||
|
||||
def call(self, inputs, embed=True):
|
||||
if embed:
|
||||
dtype = tf.keras.backend.dtype(inputs)
|
||||
if dtype != 'int32' and dtype != 'int64':
|
||||
inputs = math_ops.cast(inputs, 'int32')
|
||||
return embedding_ops.embedding_lookup(self.w, inputs)
|
||||
else:
|
||||
return tf.tensordot(inputs, tf.transpose(self.w), 1) + self.b
|
||||
|
||||
# input for the keras model
|
||||
tokens = tf.keras.layers.Input(shape=(seq_length,), dtype='int32')
|
||||
|
||||
# instantiates a tied softmax class
|
||||
tied_embedding_softmax = TiedEmbeddingSoftmax()
|
||||
|
||||
# embedded tokens, before passing it to the transformer
|
||||
embedded = tied_embedding_softmax(tokens, embed=True)
|
||||
|
||||
# the activations after passing it from the transformer
|
||||
# for some odd reason, TPUs don't play well with specifying the arguments of the Encoder() function
|
||||
# so you have to leave them at their defaults
|
||||
transformed = transformer.Encoder()(embedded, training=False)
|
||||
|
||||
|
||||
# pass the activations from our tiedsoftmax class
|
||||
# this time with embed=False denoting that we are doing the softmax operation
|
||||
# and not a lookup
|
||||
logits = tied_embedding_softmax(transformed, embed=False)
|
||||
|
||||
|
||||
# finally, define the Keras model with inputs as tokens and outputs as the logits we just computed
|
||||
model = tf.keras.Model(inputs=tokens, outputs=logits)
|
||||
|
||||
|
||||
# the loss function is a simple categorical crossentropy between the logits and the labels
|
||||
def loss(labels, logits):
|
||||
return tf.keras.losses.sparse_categorical_crossentropy(labels, logits, from_logits=True)
|
||||
|
||||
# the optimizer is not used since this code only supports inference
|
||||
# however, to compile the model, we still define it
|
||||
optimizer = tf.contrib.tpu.CrossShardOptimizer(
|
||||
tf.contrib.estimator.clip_gradients_by_norm(
|
||||
tf.train.AdagradOptimizer(learning_rate=1e-2), 0.25)
|
||||
)
|
||||
|
||||
# compile the model with the optimizer and loss
|
||||
model.compile(optimizer=optimizer, loss=loss)
|
||||
print(model.summary())
|
||||
|
||||
|
||||
# IMPORTANT
|
||||
# this is where the saved model is presented to the code
|
||||
# the model directory should have the model checkpoint and
|
||||
# a checkpoint file
|
||||
run_config = tf.contrib.tpu.RunConfig(
|
||||
model_dir=args.model_dir)
|
||||
|
||||
|
||||
# this converts the Keras model to a TensorFlow estimator
|
||||
# this step is critical
|
||||
# remember to patch the TF 1.14 file before running the code, else you're going to see errors here
|
||||
estimator_model = tf.keras.estimator.model_to_estimator(keras_model=model, config=run_config)
|
||||
|
||||
# we now create a serving function from this estimator
|
||||
# this enables us to load the model once and easily query it multiple times
|
||||
def serving_input_fn():
|
||||
inputs = {'input_1': tf.placeholder(tf.int32, [2,seq_length])}
|
||||
return tf.estimator.export.ServingInputReceiver(inputs, inputs)
|
||||
predict_fn = tf.contrib.predictor.from_estimator(estimator_model, serving_input_fn)
|
||||
|
||||
# almost there, we now take the user prompt and tokenize with BPE
|
||||
# load BPE codes
|
||||
bpe = fastBPE.fastBPE('codes', 'vocab')
|
||||
|
||||
domains = []
|
||||
with open('control_codes.txt', 'r') as f:
|
||||
domains = [line.split() for line in f.readlines()]
|
||||
domains = [(t[1], float(t[0])) for t in domains]
|
||||
|
||||
|
||||
while True:
|
||||
_prompt = raw_input('ENTER PROMPT: ') if not use_py3 else input('ENTER PROMPT: ')
|
||||
|
||||
ppls = {}
|
||||
# loop over all domains and compute perplexity
|
||||
for domain, domain_prior in domains:
|
||||
print(u'computing for domain: {}'.format(domain))
|
||||
# tokenize data and add domain tag to it
|
||||
prompt = domain + u' ' + _prompt
|
||||
split_prompt = bpe.apply([prompt])[0].split()
|
||||
|
||||
# numericalize data and pad to the seq_len dimension
|
||||
text = [word2idx[i] for i in split_prompt]
|
||||
padding_text = text + [0] * (seq_length - len(text))
|
||||
tokens_generated = np.tile(padding_text, (2,1))
|
||||
output_scores = predict_fn({'input_1':tokens_generated})['tied_embedding_softmax'].squeeze()[0]
|
||||
token_scores = output_scores[:-1]
|
||||
|
||||
# compute the perplexity for this sequence
|
||||
xent = 0
|
||||
for sequence_idx, token_idx in enumerate(text[1:]):
|
||||
token = idx2word[token_idx]
|
||||
|
||||
# compute the probability of this token
|
||||
Z = np.exp(token_scores[sequence_idx]).sum()
|
||||
token_prob = np.exp(token_scores[sequence_idx, token_idx]) / Z
|
||||
xent -= np.log(token_prob) / len(text[1:])
|
||||
|
||||
ppls[domain] = round(np.exp(xent), 6)
|
||||
#print(u'{} ppl = {}'.format(domain, ppls[domain]))
|
||||
|
||||
# sort the domains based on perplexities and print
|
||||
ppls = [(k, v) for k, v in ppls.items()]
|
||||
ppls.sort(key=lambda x: x[1])
|
||||
print('PROMPT: {}'.format(_prompt))
|
||||
for t in ppls:
|
||||
domain, ppl = t
|
||||
print(u'{} ppl = {}'.format(domain, ppl))
|
||||
@@ -0,0 +1,139 @@
|
||||
import tensorflow as tf
|
||||
import numpy as np
|
||||
|
||||
def angle_defn(pos, i, d_model_size):
|
||||
angle_rates = 1 / np.power(10000, (2 * (i//2)) / np.float32(d_model_size))
|
||||
return pos * angle_rates
|
||||
|
||||
def positional_encoding(position, d_model_size):
|
||||
# create the sinusoidal pattern for the positional encoding
|
||||
angle_rads = angle_defn(np.arange(position)[:, np.newaxis], np.arange(d_model_size)[np.newaxis, :], d_model_size)
|
||||
|
||||
sines = np.sin(angle_rads[:, 0::2])
|
||||
cosines = np.cos(angle_rads[:, 1::2])
|
||||
|
||||
pos_encoding = tf.cast(np.concatenate([sines, cosines], axis=-1)[np.newaxis, ...], dtype=tf.float32)
|
||||
return pos_encoding
|
||||
|
||||
|
||||
|
||||
def scaled_dot_product_attention(q, k, v, mask):
|
||||
# calculate attention
|
||||
matmul_qk = tf.matmul(q, k, transpose_b=True)
|
||||
|
||||
dk = tf.cast(tf.shape(k)[-1], tf.float32)
|
||||
scaled_attention_logits = matmul_qk / tf.math.sqrt(dk)
|
||||
|
||||
if mask is not None:
|
||||
scaled_attention_logits += (mask * -1e9)
|
||||
|
||||
attention_weights = tf.nn.softmax(scaled_attention_logits, axis=-1)
|
||||
output = tf.matmul(attention_weights, v)
|
||||
return output
|
||||
|
||||
class MultiHeadAttention(tf.keras.layers.Layer):
|
||||
def __init__(self, d_model_size, num_heads):
|
||||
super(MultiHeadAttention, self).__init__()
|
||||
self.num_heads = num_heads
|
||||
self.d_model_size = d_model_size
|
||||
|
||||
self.depth = int(d_model_size / self.num_heads)
|
||||
|
||||
self.Wq = tf.keras.layers.Dense(d_model_size)
|
||||
self.Wk = tf.keras.layers.Dense(d_model_size)
|
||||
self.Wv = tf.keras.layers.Dense(d_model_size)
|
||||
|
||||
self.dense = tf.keras.layers.Dense(d_model_size)
|
||||
|
||||
def split_into_heads(self, x, batch_size):
|
||||
x = tf.reshape(x, (batch_size, -1, self.num_heads, self.depth))
|
||||
return tf.transpose(x, perm=[0, 2, 1, 3])
|
||||
|
||||
def call(self, v, k, q, mask):
|
||||
batch_size = tf.shape(q)[0]
|
||||
|
||||
q = self.Wq(q)
|
||||
k = self.Wk(k)
|
||||
v = self.Wv(v)
|
||||
|
||||
q = self.split_into_heads(q, batch_size)
|
||||
k = self.split_into_heads(k, batch_size)
|
||||
v = self.split_into_heads(v, batch_size)
|
||||
|
||||
scaled_attention = tf.transpose(scaled_dot_product_attention(q, k, v, mask), perm=[0, 2, 1, 3])
|
||||
original_size_attention = tf.reshape(scaled_attention, (batch_size, -1, self.d_model_size))
|
||||
output = self.dense(original_size_attention)
|
||||
|
||||
return output
|
||||
|
||||
|
||||
|
||||
def point_wise_feed_forward_network(d_model_size, dff):
|
||||
return tf.keras.Sequential([tf.keras.layers.Dense(dff, activation='relu'),
|
||||
tf.keras.layers.Dense(d_model_size)])
|
||||
|
||||
|
||||
class EncoderLayer(tf.keras.layers.Layer):
|
||||
def __init__(self, d_model_size, num_heads, dff, rate=0.1):
|
||||
super(EncoderLayer, self).__init__()
|
||||
|
||||
self.multi_head_attention = MultiHeadAttention(d_model_size, num_heads)
|
||||
self.ffn = point_wise_feed_forward_network(d_model_size, dff)
|
||||
|
||||
self.layernorm1 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
|
||||
self.layernorm2 = tf.keras.layers.LayerNormalization(epsilon=1e-6)
|
||||
|
||||
self.dropout1 = tf.keras.layers.Dropout(rate)
|
||||
self.dropout2 = tf.keras.layers.Dropout(rate)
|
||||
|
||||
def call(self, x, training, mask):
|
||||
normed = self.layernorm1(x)
|
||||
attn_output = self.multi_head_attention(normed, normed, normed, mask)
|
||||
attn_output = self.dropout1(attn_output, training=training)
|
||||
out1 = x + attn_output
|
||||
|
||||
out2 = self.layernorm2(out1)
|
||||
ffn_output = self.ffn(out2)
|
||||
ffn_output = self.dropout2(ffn_output, training=training)
|
||||
out2 = out1 + ffn_output
|
||||
|
||||
return out2
|
||||
|
||||
|
||||
|
||||
|
||||
class Encoder(tf.keras.layers.Layer):
|
||||
def __init__(self, num_layers=48, d_model_size=1280, num_heads=16, dff=8192, input_vocab_size=50000,
|
||||
rate=0.1, **kwargs):
|
||||
super(Encoder, self).__init__()
|
||||
|
||||
self.d_model_size = d_model_size
|
||||
self.num_layers = num_layers
|
||||
|
||||
self.pos_encoding = positional_encoding(input_vocab_size, self.d_model_size)
|
||||
|
||||
for i in range(num_layers):
|
||||
setattr(self, "layer%i" % i, EncoderLayer(d_model_size, num_heads, dff, rate))
|
||||
|
||||
self.layernorm = tf.keras.layers.LayerNormalization(epsilon=1e-6)
|
||||
self.dropout = tf.keras.layers.Dropout(rate)
|
||||
|
||||
def get_config(self):
|
||||
base_config = super(Encoder, self).get_config()
|
||||
return base_config
|
||||
|
||||
def call(self, x, training):
|
||||
|
||||
seq_len = tf.shape(x)[1]
|
||||
|
||||
mask = 1 - tf.linalg.band_part(tf.ones((seq_len, seq_len)), -1, 0)
|
||||
|
||||
x *= tf.math.sqrt(tf.cast(self.d_model_size, tf.float32))
|
||||
x += self.pos_encoding[:, :seq_len, :]
|
||||
|
||||
x = self.dropout(x, training=training)
|
||||
|
||||
for i in range(self.num_layers):
|
||||
x = getattr(self, "layer%i" % i)(x, training, mask)
|
||||
return self.layernorm(x)
|
||||
|
||||
+246531
File diff suppressed because it is too large
Load Diff
Reference in New Issue
Block a user