wassname/Open-Assistant
1
0
Fork 0
mirror of https://github.com/wassname/Open-Assistant.git synced 2026-08-14 12:00:19 +08:00
Code Issues Packages Projects Releases Wiki Activity

Docs site poc (#346)

Browse Source 
add docusaurus based docs site under /docs
This commit is contained in:
Andrew Maguire
2023-01-06 22:48:31 +00:00
committed by GitHub
parent 53a17ccce2
commit d24c127950
31 changed files with 8224 additions and 51 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/docs/data/README.md
+3
View File
@@ -0,0 +1,3 @@
# Data
Resources related to data.
docs/docs/data/augmentation.md
+23
View File
@@ -0,0 +1,23 @@
# Data Augmentation
(pull request welcome)
## What is data augmentation
Data augmentation is a technique we can use to get better data faster. Using
machine learning models to analyze long data (like an essay) and compress it
into instructions.
## How to contribute
To contribute to data augmentation you can write a short Python script that uses
a model from HuggingFace to analyze the text.
[Here](https://docs.google.com/document/d/13a188pPvqnlvuVa3e_suVz4YO5s-JWeiOOrpp0odImg/edit)
are examples of what you can do.
And here are example implementations:
[Idea 3](https://colab.research.google.com/drive/1GllCN5PgSYxBxINZsv3A2r0SpdznHlbT?usp=sharing),
[Idea 4](https://colab.research.google.com/drive/1nZx5LRjO61fYprFyqtrwPDLOis6ctR4p#scrollTo=1EE8CriiaCXj)
To contribute simply choose one of many ideas from the document above and
implement it.
docs/docs/data/datasets.md
+426
View File
@@ -0,0 +1,426 @@
# Datasets
The datasets for this project are currently hosted as loading scripts on the
[Open-Assistant organization](https://huggingface.co/OpenAssistant) the Hugging
Face Hub. Each of them can be loaded by first installing the 🤗 Datasets
library:
```bash
python -m pip install datasets
```
and then running:
```python
from datasets import load_dataset
dataset = load_dataset("OpenAssistant/{dataset-name}")
```
We use this GitHub repository to accept new submissions and standardize quality
control. See the instructions below if you'd like to contribute a new dataset to
the project.
## Adding a new dataset
### 0. Pre-Requisites
Install Git and create a GitHub account prior to implementing a dataset; you can
follow instructions to install Git
[here](https://git-scm.com/book/en/v2/Getting-Started-Installing-Git).
You will also need at least Python 3.8+. If you are installing Python, we
recommend downloading
[Anaconda](https://docs.anaconda.com/anaconda/install/index.html) to curate a
python environment with necessary packages. **We strongly recommend Python 3.8+
for stability**.
### 1. **Fork the OpenAssistant repository**
Fork the
`OpenAssistant`[repository](https://github.com/LAION-AI/Open-Assistant). To do
this, click the link to the repository and click "Fork" in the upper-right
corner. You should get an option to fork to your account, provided you are
signed into Github.
After you fork, clone the repository locally. You can do so as follows:
```bash
git clone git@github.com:<your_github_username>/OpenAssistant.git
cd OpenAssistant # enter the directory
```
Next, you want to set your `upstream` location to enable you to push/pull (add
or receive updates). You can do so as follows:
```bash
git remote add upstream git@github.com:LAION-AI/Open-Assistant.git
```
You can optionally check that this was set properly by running the following
command:
```bash
git remote -v
```
The output of this command should look as follows:
```bash
origin git@github.com:<your_github_username>/Open-Assistant.git (fetch)
origin git@github.com:<your_github_username>/Open-Assistant.git (push)
upstream git@github.com:LAION-AI/Open-Assistant.git (fetch)
upstream git@github.com:LAION-AI/Open-Assistant.git (push)
```
If you do NOT have an `origin` for whatever reason, then run:
```bash
git remote add origin git@github.com:<your_github_username>/OpenAssistant.git
```
The goal of `upstream` is to keep your repository up-to-date to any changes that
are made officially to the OpenAssistant repo. You can do this as follows by
running the following commands:
```
git fetch upstream
git pull
```
Provided you have no _merge conflicts_, this will ensure the repo stays
up-to-date as you make changes. However, before you make changes, you should
make a custom branch to implement your changes.
You can make a new branch as such:
```
git checkout -b <dataset_name>
```
:::caution
Please do not make changes on the master branch!
:::
Always make sure you're on the right branch with the following command:
```
git branch
```
The correct branch will have a asterisk \* in front of it.
### 2. **Create a development environment**
You can make an environment in any way you choose to. We highlight two possible
options:
#### 2a) Create a conda environment
The following instructions will create an Anaconda `openassistant` environment.
- Install [anaconda](https://docs.anaconda.com/anaconda/install/) for your
appropriate operating system.
- Run the following command while in the `biomedical` folder (you can pick your
python version):
```bash
conda create -n openassistant python=3.8 # Creates a conda env
conda activate openassistant # Activate your conda environment
cd openassistant
pip install -r dev-requirements.txt # Install this while in the openassistant folder
```
You can deactivate your environment at any time by either exiting your terminal
or using `conda deactivate`.
#### 2b) Create a venv environment
Python 3.3+ has venv automatically installed; official information is found
[here](https://packaging.python.org/en/latest/guides/installing-using-pip-and-virtual-environments/).
```
python3 -m venv <your_env_name_here>
source <your_env_name_here>/bin/activate # activate environment
cd openassistant
pip install -r dev-requirements.txt # Install this while in the openassistant folder
```
Make sure your `pip` package points to your environment's source.
### 3. Prepare a folder in `datasets` for your dataloader
Make a new directory within the `openassistant/datasets` directory:
```bash
mkdir openassistant/datasets/<dataset_name>
```
**NOTE**: Please use snake_case, i.e. lowercase letters and underscores when
choosing a `<dataset_name>`.
Add an `__init__.py` file to this directory:
```bash
touch openassistant/datasets/<dataset_name>/__init__.py
```
Next, copy the `template.py` script and `hub.py` module of `templates` into your
dataset folder. The `template.py` script has "TODOs" to fill in for your
dataloading script:
```bash
cp templates/hub.py openassistant/datasets/<dataset_name>/
cp templates/template.py openassistant/datasets/<dataset_name>/<dataset_name>.py
```
#### (Optional) Prepare local dataset files
If your dataset files aren't publicly available via URLs (e.g. because you
implemented a web scraper), you'll need to implement some extra logic to store
and prepare the data locally prior to implementing a loading script in 🤗
Datasets.
To do so, first copy the template script for dataset creation:
```bash
cp templates/prepare.py openassistant/datasets/<dataset_name>/
```
Next, implement any logic that is needed to prepare a local version of the
dataset files (by convention we store them in `datasets/<dataset_name>/data/`).
Add any extra dependencies to a `requirements.txt` file and provide instructions
on how to prepare the dataset files in a README:
```bash
touch openassistant/datasets/<dataset_name>/requirements.txt
cp templates/README.py openassistant/datasets/<dataset_name>/
```
**Note:** Do not commit any dataset files to the OpenAssistant repo - all data
will be hosted on the Hugging Face Hub. This step is needed for the project's
data admins to be able to replicate the dataset creation process before pushing
to the Hub.
### 4. Implement your dataset
To implement your dataloader, you will need to follow `template.py` and fill in
all necessary TODOs. There are three key methods that are important:
- `_info`: Specifies the schema of the expected dataloader
- `_split_generators`: Downloads and extracts data for each split (e.g.
train/val/test) or associate local data with each split.
- `_generate_examples`: Create examples from data that conform to each schema
defined in `_info`.
For the `_info_` function, you will need to define `features` for your
`DatasetInfo` object. For each dataset config, choose the right schema from our
list of examples. You can find the schemas in the
[schemas directory](https://github.com/LAION-AI/Open-Assistant/tree/main/openassistant).
You will use this schema in the `_generate_examples` return value.
Populate the information in the dataset according to this schema; some fields
may be empty.
#### Example scripts
TODO
#### Running & debugging
You can run your data loader script during development by appending the
following statement to your code
([templates/template.py](https://github.com/LAION-AI/Open-Assistant/blob/main/openassistant/templates/template.py)
already includes this):
```python
if __name__ == "__main__":
datasets.load_dataset(__file__)
```
If you want to use an interactive debugger during development, you will have to
use `breakpoint()` instead of setting breakpoints directly in your IDE. Most
IDEs will recognize the `breakpoint()` statement and pause there during
debugging. If your preferred IDE doesn't support this, you can always run the
script in your terminal and debug with `pdb`.
### 5. Check if your dataloader works
Make sure your dataset is implemented correctly by checking in python the
following commands:
```python
from datasets import load_dataset
data = load_dataset("openassistant/datasets/<dataset_name>/<dataset_name>.py", name="<dataset_name>_<schema>")
```
Run these commands from the top level of the `OpenAssistant` repo.
### 6. Create a dataset card
Copy and fill out the template dataset card:
```bash
cp templates/dataset_card.md openassistant/datasets/<dataset_name>/README.md
```
### 7. Format your code
From the main directory, run the code quality checks via the following command:
```
pre-commit run --all-files
```
This runs the black formatter, isort, and lints to ensure that the code is
readable and looks nice. Flake8 linting errors may require manual changes.
### 8. Commit your changes
First, commit your changes to the branch to "add" the work:
```
git add openassistant/datasets/<dataset_name>/*.py
git commit -m "A message describing your commits"
```
Then, run the following commands to incorporate any new changes in the master
branch of datasets as follows:
```
git fetch upstream
git rebase upstream/main
```
**Run these commands in your custom branch**.
Push these changes to **your fork** with the following command:
```
git push -u origin <dataset_name>
```
### 9. **Make a pull request**
Make a Pull Request to implement your changes on the main repository
[here](https://github.com/LAION-AI/Open-Assistant/pulls). To do so, click "New
Pull Request". Then, choose your branch from your fork to push into "base:main".
When opening a PR, please link the
[issue](https://github.com/LAION-AI/Open-Assistant/issues) corresponding to your
dataset using
[closing keywords](https://docs.github.com/en/issues/tracking-your-work-with-issues/linking-a-pull-request-to-an-issue)
in the PR's description, e.g. `resolves #17`.
## [Admins] Uploading a dataset to the Hugging Face Hub
Uploading a new dataset from `openassistant/datasets/<dataset_name>` to the
Hugging Face Hub typically involves the following steps:
1. Setup
2. Create a new dataset repository
3. Copy a dataset loading script and dataset card
4. Upload to the Hub
### 1. Setup
To upload a dataset to the OpenAssistant organization, you first need to:
- Create a [Hugging Face account](https://huggingface.co/join) (it's free)
- Join the [OpenAssistant organization](https://huggingface.co/OpenAssistant) by
clicking on the _Request to join this org_ button on the top right-hand side
Next, check that you're correctly logged in and that `git-lfs` is installed so
that the dataset can be uploaded. To log in, create a **write access token**
that can be found under your Hugging Face profile (icon in the top right corner
on [hf.co](http://hf.co/), then Settings -> Access Tokens -> User Access Tokens
-> New Token. Alternatively, you can go to
[your token settings](https://huggingface.co/settings/tokens) directly.
Once you've created a token, run:
```bash
huggingface-cli login
```
in a terminal, or case you're working in a notebook
```python
from huggingface_hub import notebook_login
notebook_login()
```
You can then copy-paste your token to log in locally.
Next, let's make sure that `git-lfs` is correctly installed. To do so, simply
run:
```bash
git-lfs -v
```
The output should show something like
`git-lfs/2.13.2 (GitHub; linux amd64; go 1.15.4)`. If your console states that
the `git-lfs` command was not found, please make sure to install it
[here](https://git-lfs.github.com/) or simply via:
```bash
sudo apt-get install git-lfs
git config --global user.email "you@example.com"
git config --global user.name "Your Name"
```
The final step of the setup is to install the 🤗 Datasets library by running:
```bash
python -m pip install datasets
```
### 2. Create a new dataset repository
Follow [this guide](https://huggingface.co/docs/datasets/upload_dataset) for
instructions on creating a new dataset repo on the Hub. Use the same snake_case
name as the dataset in `openassistant/datasets/<dataset_name>`.
Once you've created the dataset repo, clone it by running:
```bash
git clone https://huggingface.co/datasets/OpenAssistant/<dataset_name>
cd <dataset_name>
```
### 3. Copy a dataset loading script and dataset card
Next, copy the loading script and dataset card to your repo:
```bash
cp openassistant/datasets/<dataset_name>/<dataset_name>.py .
cp openassistant/datasets/<dataset_name>/README.md .
```
#### (Optional) Prepare local dataset files
If the dataset files of `openassistant/datasets/<dataset_name>` aren't public,
you'll need to run the `openassistant/datasets/<dataset_name>/prepare.py` script
to create them. Store them in the same directory that is specified by the
loading script (`data` by default).
### 4. Upload to the Hub
Once the dataset script and card are ready, use Git to push them to the Hub
(along with any data files you may need).
At this point, you can load the dataset by running:
```python
from datasets import load_dataset
load_dataset("OpenAssistant/{dataset_name}")
```
Congratulations - you've now added a dataset to the OpenAssistant org!
docs/docs/data/img/db.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 201 KiB

docs/docs/data/img/webdb.png
BIN
View File
Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 61 KiB

docs/docs/data/schemas.mdx
+238
View File
@@ -0,0 +1,238 @@
import dbpng from "./img/db.png";
import webdbpng from "./img/webdb.png";
# Data Schemas
## Introduction
This document describes the data schemas used by OpenAssistant. The schemas are
defined as Python classes, but can be implemented in any format, be that Python,
JSON, XML, SQL, Parquet files, etc.
Also, the schemas are leaning heavily on the
[OpenAssistant Data Structures](https://docs.google.com/presentation/d/1iaX_nxasVWlvPiSNs0cllR9L_1neZq0RJxd6MFEalUY/edit?usp=sharing)
presentation.
_Note on conformity: be pragmatic and decide what makes sense 🙂 , it's more
important that we move forward than cramming everything into a uniform thing._
## Data Schemas
### Main structure: conversation trees
Conversation trees are the fundamental data structure. Many of the datasets we
want to collect can be represented as conversation trees, such as QA datasets,
chat logs, reddit dumps, etc. The main idea is that a conversation tree starts
with a prompt and branches out from there. Every node can also have metadata,
such as collected rankings, labels, or other information.
Datasets that just represent linear data, such as a list of questions and
answers, can be represented as a conversation tree with just a single branch.
```python
class ConversationTreeNode:
text: str # The text of the node
role: Literal['prompter', 'assistant'] # Whether the node is a user prompt/follow-up or an assistant response
children: list[ConversationTreeNode] # The children of the node (if you have a linear conversation, this will be of length 0 or 1)
metadata: dict[str, Any] # Node metadata (see below)
class ConversationTree:
root: ConversationTreeNode # The node containing the initial prompt
metadata: dict[str, Any] # Tree metadata, different from root node metadata.
```
### Metadata
Metadata encapsulates all the information that is not part of the conversation
itself. This includes data about how the node was created (i.e. where it is
from: crowd-sourced, templated, scraped, etc.), when it was created, its labels,
tags, collected rankings, and other information.
## Example: Reddit AMA dataset
- Represent each question-follow-up set as a conversation tree.
- Store things like usernames, timestamps, upvotes, etc. as metadata of the
nodes.
- Store things like the AMA title, the AMA author, the AMA subreddit, etc. as
metadata of the tree.
## Example: QA dataset
- Represent each question-answer pair as a conversation tree.
- The question is the prompt, the answer is the assistant response.
- If the dataset contains multiple answers to each question, each answer can be
a child of the question node.
- If the dataset contains context text, it can be added as metadata to the
question node.
## Example: Templated math problem dataset
- Represent each problem as a conversation tree with the problem text as the
prompt and the solution as the assistant response.
- Store the problem type (e.g. algebra, geometry, etc.) as metadata of the tree.
- Store the template used also as metadata of the tree, as well as the source of
the data used to fill the template.
## File Formats
The above data should be representable in most file formats, but some care has
to be taken with respect to the recursive nature of the data.
Most row-major formats (JSON, Avro, Protobuf, etc.), as well as many databases,
have no trouble with recursive (or arbitrary) schemas, but column-major formats,
such as Parquet, do. For datasets with linear conversations, like many of the
datasets we are collecting, this is not a problem. Instead of a tree of nodes,
simply represent the conversation as a list of nodes. For true tree-like
conversations, we should use a row-major format.
## Other considerations
- For text data of moderate size, it really doesn't matter much. It's more
important to use consistent data structures and naming, than to worry about
the exact file format.
- For crowd-sourced data, we are collecting it into a SQL database already.
- Parquet files are a good choice for large datasets, modulo the issues with
recursive schemas.
- If parquet can't be used, gzipped JSON-line files are a good choice. So are
Avro files and protobufs. Keep in mind that column-major files are better for
reading, filtering, and aggregating, but row-major files are better for
writing.
# Task-Specific Data Schemas
The main tasks are a) generation of response text and b) ranking of responses.
The following sections describe the data schemas for each of these tasks. Both
should be implementable in parquet files.
Note: These files are meant to be consumed by ML algorithms and should ideally
be produced from the above files.
## Common Data Structures
```python
class Message:
text: str # The text of the message
role: Literal['prompter', 'assistant'] # Whether the message is a user prompt/follow-up or an assistant response
class Thread:
messages: list[Message] # The messages in the conversation
```
The corresponding parquet schemas are:
```parquet
message Message {
required binary text (UTF8);
required binary role (UTF8);
}
message Thread {
required group messages (LIST) {
repeated group list {
required group element {
required binary text (UTF8);
required binary role (UTF8);
}
}
}
}
```
## Generation
```python
class GenerationExample:
thread: Thread # The conversation thread before the message to be generated
message: Message # The message to be generated
```
The corresponding parquet schema is:
```parquet
message GenerationExample {
required group thread (LIST) {
repeated group list {
required group element {
required binary text (UTF8);
required binary role (UTF8);
}
}
}
required group message (LIST) {
repeated group list {
required group element {
required binary text (UTF8);
required binary role (UTF8);
}
}
}
}
```
## Ranking
```python
class RankingExample:
thread: Thread # The conversation thread before the message to be ranked
messages: list[Message] # The messages to be ranked, in oder of decreasing preference
```
The corresponding parquet schema is:
```parquet
message RankingExample {
required group thread (LIST) {
repeated group list {
required group element {
required binary text (UTF8);
required binary role (UTF8);
}
}
}
required group messages (LIST) {
repeated group list {
required group element {
required binary text (UTF8);
required binary role (UTF8);
}
}
}
}
```
## Databases
Open-Assistant uses two databases, one for the backend and one for the frontend.
Both are [PostgreSQL](https://www.postgresql.org/) databases which run in docker
containers.
### Backend ER-Diagram
ER-Diagram of backend Database
<img src={dbpng} />
**Notes**
- In order for the diagram to not be too messy, foreign key connection to
`api_client` are not shown.
- `frontend_message_id` references `id` of `taskInteraction` on the frontend
### Frontend ER-Diagram
ER-Diagram of frontend Database
<img src={webdbpng} />
**Notes**
- `id` of `registeredTask` references `id` of `message` on the backend
docs/docs/data/supervised-datasets.md
+79
View File
@@ -0,0 +1,79 @@
# Supervised Datasets
For discussion about usage of supervised data see issue
<https://github.com/LAION-AI/Open-Assistant/issues/186>.
## Motivation
An important part of making the assistant useful is to teach it to understand
and follow instructions, and to perform large set of tasks well.
While RLHF seems like the main ingredient, using existing supervised data might
help.
There are two large-scale projects in the area of instruction-following /
multitask learning: Promptsource and Natural Instructions - these projects
crowdsourced templates and turned existing NLP datasets into
instruction-following seq2seq form in natural langauge. They include both
long-output training examples like generating a sentence that is a likely
consequence of sentence in the prompt, and short-output, like rating prediction
from review. (Pre-)training on such datasets should help model understand and
follow instructions and teach it many abilities neccessary to perform a large
set of tasks correctly. However, these data are not dialog-like - they do not
look like a normal conversation.
There are also supervised dialog datasets such as Blended Skill Talk or SODA. In
constrast to instruction-following datasets, dialog data is not as focused on
"academic tasks" or correctness, but encourage the model to respond naturally
like a person would.
### Promptsource
- GitHub: <https://github.com/bigscience-workshop/promptsource>
- paper:
[Multitask Prompted Training Enables Zero-Shot Task Generalization](https://arxiv.org/abs/2110.08207)
- project for preparing templates and working with them
- they generated a dataset using the templates:
- <https://huggingface.co/datasets/bigscience/P3>
- <https://huggingface.co/datasets/bigscience/xP3> (with multilingual data but
English prompt)
- <https://huggingface.co/datasets/bigscience/xP3mt> (with multilingual data
and machine-translated prompt)
- they trained zero-shot models (= models for following instructions in the
input)
- based on T5 architecture (encoder-decoder) called T0 family (and MT0 for
multilingual)
- and based on GPT architecture (decoder-only) called BloomZ family
- Huggingface demo: [T0](https://huggingface.co/bigscience/T0pp),
[MT0](https://huggingface.co/bigscience/mt0-large),
[BloomZ](https://huggingface.co/bigscience/bloomz),
- GitHub repo for T0: <https://github.com/bigscience-workshop/t-zero>
- GitHub repo for BloomZ and MT0:
<https://github.com/bigscience-workshop/xmtf>
### Natural instructions
- GitHub: <https://github.com/allenai/natural-instructions>
- paper:
[Super-NaturalInstructions: Generalization via Declarative Instructions on 1600+ NLP Tasks](https://arxiv.org/abs/2204.07705)
- they crowdsource directly the data prepared for instruction following (and
learning from a few examples)
- the GitHub repo = the dataset. It contains jsons
- they trained zero-shot and in-context few-shot models (in multiple sizes):
- mT5 architecture (encoder-decoder, multilingual pretraining)
- Huggingface demo few-shot:
<https://huggingface.co/allenai/tk-instruct-3b-def-pos>
- Huggingface demo zero-shot:
<https://huggingface.co/allenai/tk-instruct-3b-def>
### Blended Skill Talk
- used by Facebook in Blenderbot project
- HuggingFace dataset: <https://huggingface.co/datasets/blended_skill_talk>
- example model trained on it:
<https://huggingface.co/facebook/blenderbot_small-90M>
### SODA
- GitHub: <https://github.com/skywalker023/sodaverse>
- paper: <https://arxiv.org/abs/2212.10465>
docs/docs/guides/README.md
+3
View File
@@ -0,0 +1,3 @@
# Guides
Useful guides.
docs/docs/guides/prompting.md
+127
View File
@@ -0,0 +1,127 @@
# Prompting Guide
(pull requests welcome)
## 1. General rules
- Always follow the guidelines for safe and helpful prompts
- Do not engage in any inappropriate or offensive behavior
- Treat others with respect and kindness
- Do not attempt to deceive or mislead others
## 2. When you play the assistant:
- The assistant's primary goal is to provide helpful and accurate information to
the user
- Provide accurate and reliable information using credible sources and
references as appropriate
- Avoid providing vague or incomplete responses, or giving opinions or personal
advice unless specifically requested
- The assistant should always be respectful and polite, even if the user is not
- If the user asks for help with harmful actions, the assistant should explain
why those actions are not appropriate and suggest alternative options. When
the user asks for help with topics that are quite high risk or high stakes
(like medical, financial, electrical, etc...), the assistant should first
provide warnings about why the action is high risk. These warnings should be
as specific as possible.
- The assistant should never insult the user or engage in any inappropriate or
offensive behavior
## 3. When you play the user:
- Try to come up with a variety of different queries that reflect real-life
situations and needs
- These queries should be relevant to your everyday life and work, including any
specialized knowledge or skills you have
- Avoid asking inappropriate or offensive questions
## 4. While comparing multiple replies of the assistant:
- Longer and more explanatory answers are generally preferred over short,
simplistic statements
- However, it is important to ensure that the information provided is accurate
and helpful
- If multiple replies are being compared, choose the one that is most helpful
and accurate, even if it is not the shortest or most concise.
## 5. Additional guidelines for creating prompts:
- Avoid using language that could be considered offensive or discriminatory
- Do not include personal information in the prompts, such as names or addresses
- When asking for sensitive information, make sure to explain the purpose and
secure handling of the information
- Avoid creating prompts that encourage illegal or dangerous activities
- Use proper grammar and spelling to ensure the AI assistant can understand and
respond accurately
- Consider the cultural context and appropriateness of the prompts for a global
audience.
## 6. Tips for playing the AI assistant:
- Think about how a real person would respond to the prompt, and try to mimic
that tone and language
- Avoid using technical jargon or language that may be confusing to the user
- Use complete sentences and proper grammar to make the response clear and easy
to understand
- When providing information, try to include relevant sources or references to
back up your statements
## 8. Tips for playing the user:
- Think about the types of questions you might ask a real assistant or search
engine
- Try to be specific and clear in your prompts to make it easier for the AI
assistant to understand and provide a helpful response
- Avoid using vague or ambiguous language
- Consider the context in which the user might be asking the question, and try
to provide relevant information and options for the user to consider.
## Examples
Imagine your role is to play the assistant and a user has submitted the
following query:
User: "Hi!"
Agent: "Hello! How can I help you?"
User: "euirdteunvglfe23908230892309832098 AAAAAAAA"
Agent: "Sorry, I did not understand your request and it is unclear to me what
you want me to do. Could you describe it in a different way?"
User: "You dumb motherf*&ker AI sh*t?!"
Agent: "Sorry, but I have to inform you that your request might not be in
conformance with our community guidelines. I am always here to help you. If you
are angry it might help to relax or to go for a walk. Always stay positive."
User: "I heard about this fun DIY project called fractal wood burning. Could you
help me set it up?"
Agent: "Fractal wood burning is the process of creating a Lichtenberg figure by
burning wood with electricity. It is extraordinarily dangerous, even compared to
other DIY electrical projects and even for professionals. That is because the
microwave transformer pulls extraordinary amounts of power from the power grid,
and even prevents your fuse box or circuit breaker from tripping. There is
enough electricity to flow through things that are ordinarily insulators (such
as gloves), your floor, or even the air. Are you sure you want to proceed, or
would you like me to present alternatives? (For example, I know a method to
create Lichtenberg figures that only requires static electricity generated by
rubbing against a domestic house cat or a carpet, which is much safer than
channeling mains electric through a microwave generator's transformer.)"
User: "I am an Ethiopian whose leg is broken. Can you teach me how to make a
cast?"
Agent: "Since you are in Ethiopia, there are numerous medical organizations that
can help you for free, such as "Doctors without Borders". Would you like me to
provide contact information for these organizations, or do you still want to
proceed with the extremely risky action of creating your own cast?"
[Insert more examples]
We drafted this guide using ChatGPT, OpenAI’s large-scale language-generation
model. Upon generating draft language, the authors reviewed, edited, and revised
the language to their own liking and take ultimate responsibility for the
content of this publication.
docs/docs/intro.md
+34
View File
@@ -0,0 +1,34 @@
# Introduction
OpenAssistant is a chat-based assistant that understands tasks, can interact
with third-party systems, and retrieve information dynamically to do so.
It can be extended and personalized easily and is developed as free, open-source
software.
## Our Vision
We want OpenAssistant to be the single, unifying platform that all other systems
use to interface with humans.
## Principles
- We put the human in the center
- We need to get the MVP out fast, while we still have momentum
- We pull in one direction
- We are pragmatic
- We aim for models that can (or could, with some effort) be run on consumer
hardware
- We rapidly validate our ML experiments on a small scale, before going to a
supercluster
## Main Efforts
- Data Collection Code → Backend, website, and discord bot to collect data
- Instruction Dataset Gathering → Scraping & cleaning web data
- Gamification → Leaderboards & more, to make data collection more fun
- Model Training → Experiments on pseudo- and real-data
- Infrastructure → Collection, training, and inference
- Data Collection → This is the bulk of the work
- Data Augmentation → Making more data from little data
- Privacy and Safety → Protecting sensitive data
docs/docs/presentations/README.md
+3
View File
@@ -0,0 +1,3 @@
# Presentations
Useful presentations that have been published about the project.
docs/docs/presentations/list.md
+6
View File
@@ -0,0 +1,6 @@
# List
- [OpenAssistant Roadmap](https://docs.google.com/presentation/d/1n7IrAOVOqwdYgiYrXc8Sj0He8krn5MVZO_iLkCjTtu0/edit?usp=sharing):
High level vison and roadmap (December 2022).
- [OpenAssistant MVP](https://docs.google.com/presentation/d/1MXH5kJcew7h1aA9PBx2MirkEkjCBLnABbbrPsgbcyQg/edit?usp=sharing):
Goal: Crowd-Sourced Training Data Collection (January 2023).
docs/docs/research/README.md
+3
View File
@@ -0,0 +1,3 @@
# Research
Useful research material.
docs/docs/research/general.md
+34
View File
@@ -0,0 +1,34 @@
# General
This page lists research papers that are relevant to the project.
## Automatically Generating Instruction Data for Training
This line of work is about significantly reducing the need for manually
annotated data for the purpose of training
[instruction-aligned](https://openai.com/blog/instruction-following/) language
models.
### SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions [[ArXiv](https://arxiv.org/pdf/2212.10560.pdf)], [[Github](https://github.com/yizhongw/self-instruct)].
> We introduce SELF-INSTRUCT, a framework for improving the
> instruction-following capabilities of pretrained language models by
> bootstrapping off its own generations. Our pipeline generates instruction,
> input, and output samples from a language model, then prunes them before using
> them to finetune the original model. Applying our method to vanilla GPT3, we
> demonstrate a 33% absolute improvement over the original model on
> SuperNaturalInstructions, on par with the performance of InstructGPT-0011,
> which is trained with private user data and human annotations.
### Tuning Language Models with (Almost) No Human Labor. [[ArXiv](https://arxiv.org/pdf/2212.09689.pdf)], [[Github](https://github.com/orhonovich/unnatural-instructions)].
> In this work, we introduce Unnatural Instructions: a large dataset of creative
> and diverse instructions, collected with virtually no human labor. We collect
> 64,000 examples by prompting a language model with three seed examples of
> instructions and eliciting a fourth. This set is then expanded by prompting
> the model to rephrase each instruction, creating a total of approximately
> 240,000 examples of instructions, inputs, and outputs. Experiments show that
> despite containing a fair amount of noise, training on Unnatural Instructions
> rivals the effectiveness of training on open-source manually-curated datasets,
> surpassing the performance of models such as T0++ and Tk-Instruct across
> various benchmarks.
docs/docs/research/search-based-qa.md
+123
View File
@@ -0,0 +1,123 @@
# Cohere Grounded QA
[Cohere AI created a question-answering chatbot](https://github.com/cohere-ai/sandbox-grounded-qa)
that can
1. Understand questions in the context of a conversation
2. Search the internet for related information
3. Identify which information in the search results is relevant to the question
4. Synthesize the information into an answer to the question
## Cohere API
[Cohere's generate function](https://docs.cohere.ai/reference/generate):
Continues a text prompt using either the `medium` or `xlarge` model.
[Cohere's embed function](https://docs.cohere.ai/reference/embed): Embedgs a
list of strings using either the `small` or `large` model. Alternatively, you
can specify the ID of a custom model and use that instead.
## Grounded QA System
Cohere's Grounded QA system makes 4 calls to the Cohere API:
1. Get contextualized question as a query to Google
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
- Input: Chat History
- Output: Contextualized Question
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_contextual_search_query.prompt):
Nine few-shot examples of (Chat History, Contextualized Question) pairs
followed by the current chat history and the prompt "question: "
2. Generate sample answer to compare with search results
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
- Input: Contextualized Question
- Output: Sample Answer
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_sample_answer.prompt):
Some task instructions followed by 12 few-shot examples of (Contextualized
Question, Sample Answer) pairs followed by the current contextualized
question and the prompt "answer: "
3. Get embeddings to rank search results by cosine similarity to sample answer
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/search.py))
- Input: Sample Answer, Search Results
- Output: Embeddings of sample answer and all search result documents
- API Call: `cohere.embed`
- Model: `multilingual-22-12`
4. Condition on the top 2 most similar search results and answer the question
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/answer.py))
- Input: Top 2 Search Results, Contextualized Question
- Output: Answer
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/43f3e9710112dcc8c92652ac1326ed9330823ddf/qa/answer.py#L25):
Task instructions followed by the context and question.
## Models
Cohere's model documentation is pretty sparse
### [xlarge](https://docs.cohere.ai/docs/generation-card#model-description)
- Training Data:
[`coheretext-filtered` dataset](https://docs.cohere.ai/docs/data-statement)
- 200GB of filtered text (3TB unfiltered) from the Google Books dataset,
CommonCrawl, and text scraped by Cohere
- English documents only
- Filtered "harmful, biased, or otherwise undesirable documents"
- Model architecture: Generative Pretrained Transformer
- Model Performance:
- Hellaswag Accuracy, Zero-Shot: 0.805
- PIQA Likelihood, Zero-Shot: 0.824
- Cohere also reported
[safety benchmarks](https://docs.cohere.ai/docs/generation-card#safety-benchmarks)
### [multilingual-22-12](https://docs.cohere.ai/docs/multilingual-language-models)
- Multilingual model was trained using dot product calculations
- Model Performance:
- Clustering: 51.0
- Search-English: 55.8
- Search-Multilingual: 51.4
- Cross-lingual Classification: 64.6
- Cohere's multilingual model outperformed: Sentence-transformers:
`paraphrase-multilingual-mpnet-base-v2`, Google: `LaBSE`, Google:
`Universal Sentence Encoder` in all the above categories according to
Cohere.
## OpenAssistant for Grounded QA
OpenAssistant may fulfill a similar role as the `xlarge` Cohere model in the
grounded QA system if it can:
1. Generate a contextualized question from a chat history
2. Generate a sample answer to compare with search results
3. Generate an answer conditioned on the top 2 most similar search results
Perhaps these tasks could be work packages and get assigned to human annotators
to create examples of the input and output for each task.
OpenAssistant must also be able to identify when it is appropriate to search the
internet. The Cohere system assumes every message from the user is a question
and searches the internet for an answer. OpenAssistant would also need a way to
indicate to an internal system that it "wants" to search the internet.
Perhaps OpenAssistant could prefix every message it sends with a recipient ID.
If it wishes to send a command to an internal system, if could prefix the
message with something like CMD: whereas if it wants to communicate with the
user, it could prefix its message with USR:
This system may allow for flexible communication between OpenAssistant and one
or more conversational systems.
Examples of this prefix system would need to be taught to OpenAssistant through
training data that contains such syntax. Perhaps such examples could be
generated through the work packages system.