wassname/Open-Assistant
1
0
Fork 0
mirror of https://github.com/wassname/Open-Assistant.git synced 2026-06-27 16:10:30 +08:00
Code Issues Packages Projects Releases Wiki Activity

run prettier with new params

Browse Source
This commit is contained in:
Gareth Davidson
2023-01-01 20:57:02 +00:00
parent 4c7b8cfd35
commit c3c7a1701a
21 changed files with 448 additions and 208 deletions
Download Patch File Download Diff File Expand all files Collapse all files
.github/workflows/build-frontend.yaml
+2 -2
View File
@@ -12,7 +12,7 @@ on:
workflow_call:
jobs:
build-frontend:
build-frontend:
runs-on: ubuntu-latest
defaults:
run:
@@ -22,7 +22,7 @@ jobs:
- uses: actions/setup-node@v3
with:
node-version: 16.x
cache: 'npm'
cache: "npm"
cache-dependency-path: website/package-lock.json
- run: npm ci
- run: npm run build
README.md
+86 -37
View File
@@ -1,12 +1,17 @@
# Open-Assistant
Open Assistant is a project meant to give everyone access to a great chat based large language model.
Open Assistant is a project meant to give everyone access to a great chat based
large language model.
We believe that by doing this we will create a revolution in innovation in language. In the same way that stable-diffusion helped the world make art and images in new ways we hope Open Assistant can help improve the world by improving language itself.
We believe that by doing this we will create a revolution in innovation in
language. In the same way that stable-diffusion helped the world make art and
images in new ways we hope Open Assistant can help improve the world by
improving language itself.
## Do you want to try it out?
If you are interested in taking a look at the current state of the project, You can set up an entire stack needed to run **Open-Assistant**, including the
If you are interested in taking a look at the current state of the project, You
can set up an entire stack needed to run **Open-Assistant**, including the
website, backend, and associated dependent services.
To start the demo, Run this in the root directory of the repository:
@@ -15,23 +20,44 @@ To start the demo, Run this in the root directory of the repository:
docker compose up --build
```
Then, navigate to `http://localhost:3000` (It may take some time to boot up) and interact with the website.
Then, navigate to `http://localhost:3000` (It may take some time to boot up) and
interact with the website.
**Note:** When logging in via email, navigate to `http://localhost:1080` to get the magic email login link.
**Note:** When logging in via email, navigate to `http://localhost:1080` to get
the magic email login link.
## The Plan
We want to get to an initial MVP as fast as possible, by following the 3-steps outlined in the InstructGPT paper.
We want to get to an initial MVP as fast as possible, by following the 3-steps
outlined in the InstructGPT paper.
1. Collect high-quality human generated Instruction-Fulfillment samples (prompt + response), goal >50k. We design a crowdsourced process to collect and reviewed prompts. We do not want to train on flooding/toxic/spam/junk/personal information data. We will have a leaderboard to motivate the community that shows progress and the most active users. Swag will be given to the top-contributors.
2. For each of the collected prompts we will sample multiple completions. Completions of one prompt will then be shown randomly to users to rank them from best to worst. Again this should happen crowd-sourced, e.g. we need to deal with unreliable potentially malicious users. At least multiple votes by independent users have to be collected to measure the overall agreement. The gathered ranking-data will be used to train a reward model.
3. Now follows the RLHF training phase based on the prompts and the reward model.
1. Collect high-quality human generated Instruction-Fulfillment samples
(prompt + response), goal >50k. We design a crowdsourced process to collect
and reviewed prompts. We do not want to train on
flooding/toxic/spam/junk/personal information data. We will have a
leaderboard to motivate the community that shows progress and the most active
users. Swag will be given to the top-contributors.
2. For each of the collected prompts we will sample multiple completions.
Completions of one prompt will then be shown randomly to users to rank them
from best to worst. Again this should happen crowd-sourced, e.g. we need to
deal with unreliable potentially malicious users. At least multiple votes by
independent users have to be collected to measure the overall agreement. The
gathered ranking-data will be used to train a reward model.
3. Now follows the RLHF training phase based on the prompts and the reward
model.
We can then take the resulting model and continue with completion sampling step 2 for a next iteration.
We can then take the resulting model and continue with completion sampling step
2 for a next iteration.
## The Vision
We are not going to stop at replicating ChatGPT. We want to build the assistant of the future, able to not only write email and cover letters, but do meaningful work, use APIs, dynamically research information, and much more, with the ability to be personalized and extended by anyone. And we want to do this in a way that is open and accessible, which means we must not only build a great assistant, but also make it small and efficient enough to run on consumer hardware.
We are not going to stop at replicating ChatGPT. We want to build the assistant
of the future, able to not only write email and cover letters, but do meaningful
work, use APIs, dynamically research information, and much more, with the
ability to be personalized and extended by anyone. And we want to do this in a
way that is open and accessible, which means we must not only build a great
assistant, but also make it small and efficient enough to run on consumer
hardware.
### Slide Decks
@@ -41,15 +67,20 @@ We are not going to stop at replicating ChatGPT. We want to build the assistant
## How can you help?
All open source projects begins with people like you. Open source is the belief that if we collaborate we can together gift our knowledge and technology to the world for the benefit of humanity.
All open source projects begins with people like you. Open source is the belief
that if we collaborate we can together gift our knowledge and technology to the
world for the benefit of humanity.
## I’m in! Now what?
[Join the OpenAssistant Contributors Discord Server!](https://ykilcher.com/open-assistant-discord), this is for work coordination.
[Join the OpenAssistant Contributors Discord Server!](https://ykilcher.com/open-assistant-discord),
this is for work coordination.
[Join the LAION Discord Server!](https://discord.com/invite/mVcgxMPD7e), it has a dedicated channel and is more public.
[Join the LAION Discord Server!](https://discord.com/invite/mVcgxMPD7e), it has
a dedicated channel and is more public.
[and / or the YK Discord Server](https://ykilcher.com/discord), also has a dedicated, but not as active, channel.
[and / or the YK Discord Server](https://ykilcher.com/discord), also has a
dedicated, but not as active, channel.
[Visit the Notion](https://ykilcher.com/open-assistant)
@@ -57,15 +88,16 @@ All open source projects begins with people like you. Open source is the belief
We have a growing task list
[of issues](https://github.com/LAION-AI/Open-Assistant/issues). Find an issue
that appeals to you and make a comment that you'd like to work on it. Include
in your comment a brief description of how you'll solve the problem and if
there are any open questions you want to discuss. Once a project coordinator
has assigned the issue to you, start working on it.
that appeals to you and make a comment that you'd like to work on it. Include in
your comment a brief description of how you'll solve the problem and if there
are any open questions you want to discuss. Once a project coordinator has
assigned the issue to you, start working on it.
If the issue is currently unclear but you are interested, please post in
Discord and someone can help clarify the issue with more detail.
If the issue is currently unclear but you are interested, please post in Discord
and someone can help clarify the issue with more detail.
**Always Welcome:** Documentation markdowns in `docs/`, docstrings, diagrams of the system architecture, and other documentation.
**Always Welcome:** Documentation markdowns in `docs/`, docstrings, diagrams of
the system architecture, and other documentation.
### Submitting Work
@@ -73,8 +105,8 @@ We're all working on different parts of Open Assistant together. To make
contributions smoothly we recommend the following:
1. [Fork this project repository](https://docs.github.com/en/get-started/quickstart/fork-a-repo)
and clone it to your local machine.
(Read more [About Forks](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks))
and clone it to your local machine. (Read more
[About Forks](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/about-forks))
1. Before working on any changes, try to
[sync the forked repository](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/working-with-forks/syncing-a-fork)
to keep it up-to-date with the upstream repository.
@@ -83,7 +115,8 @@ contributions smoothly we recommend the following:
simplifies life for reviewers.
1. Package up a small bit of work that solves part of the problem
[into a Pull Request](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/creating-a-pull-request-from-a-fork)
and [send it out for review](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/requesting-a-pull-request-review).
and
[send it out for review](https://docs.github.com/en/pull-requests/collaborating-with-pull-requests/proposing-changes-to-your-work-with-pull-requests/requesting-a-pull-request-review).
1. If you're lucky, we can merge your change into `main` without any problems.
If there's changes to files you're working on, resolve them by:
1. First try rebase as suggested
@@ -108,20 +141,27 @@ addressed now, or filing an issue to handle it later.
## Developer Setup
Work is organized in the [project board](https://github.com/orgs/LAION-AI/projects/3).
Work is organized in the
[project board](https://github.com/orgs/LAION-AI/projects/3).
**Anything that is in the `Todo` column and not assigned, is up for grabs. Meaning we'd be happy for anyone to do these tasks.**
**Anything that is in the `Todo` column and not assigned, is up for grabs.
Meaning we'd be happy for anyone to do these tasks.**
If you want to work on something, assign yourself to it or write a comment that you want to work on it and what you plan to do.
If you want to work on something, assign yourself to it or write a comment that
you want to work on it and what you plan to do.
- To get started with development, if you want to work on the backend, have a look at `scripts/backend-development/README.md`.
- If you want to work on any frontend, have a look at `scripts/frontend-development/README.md` to make a backend available.
- To get started with development, if you want to work on the backend, have a
look at `scripts/backend-development/README.md`.
- If you want to work on any frontend, have a look at
`scripts/frontend-development/README.md` to make a backend available.
There is also a minimal implementation of a frontend in the `text-frontend` folder.
There is also a minimal implementation of a frontend in the `text-frontend`
folder.
We are using Python 3.10 for the backend.
Check out the [High-Level Protocol Architecture](https://www.notion.so/High-Level-Protocol-Architecture-6f1fd3551da74213b560ead369f132dc)
Check out the
[High-Level Protocol Architecture](https://www.notion.so/High-Level-Protocol-Architecture-6f1fd3551da74213b560ead369f132dc)
### Website
@@ -129,16 +169,25 @@ The website is built using Next.js and is in the `website` folder.
### Pre-commit
Install `pre-commit` and run `pre-commit install` to install the pre-commit hooks.
Install `pre-commit` and run `pre-commit install` to install the pre-commit
hooks.
In case you haven't done this, have already committed, and CI is failing, you can run `pre-commit run --all-files` to run the pre-commit hooks on all files.
In case you haven't done this, have already committed, and CI is failing, you
can run `pre-commit run --all-files` to run the pre-commit hooks on all files.
### Deployment
Upon making a release on GitHub, all docker images are automatically built and pushed to ghcr.io. The docker images are tagged with the release version, and the `latest` tag. Further, the ansible playbook in `ansible/dev.yaml` is run to automatically deploy the built release to the dev machine.
Upon making a release on GitHub, all docker images are automatically built and
pushed to ghcr.io. The docker images are tagged with the release version, and
the `latest` tag. Further, the ansible playbook in `ansible/dev.yaml` is run to
automatically deploy the built release to the dev machine.
### Problems and Solutions
- **I am on Ubuntu and getting `ERROR: The Compose file is invalid because:Service backend has neither an image nor a build context specified. At least one must be provided.`**
- **I am on Ubuntu and getting
`ERROR: The Compose file is invalid because:Service backend has neither an image nor a build context specified. At least one must be provided.`**
Make sure you have an up-to-date version of docker installed, and also install `docker-compose-plugin`. See [here](https://github.com/LAION-AI/Open-Assistant/issues/208) for more details.
Make sure you have an up-to-date version of docker installed, and also install
`docker-compose-plugin`. See
[here](https://github.com/LAION-AI/Open-Assistant/issues/208) for more
details.
backend/README.md
+5 -2
View File
@@ -2,7 +2,9 @@
## REST Server Configuration
Please either use environment variables or create a `.env` file in the backend root directory (in which this readme file is located) to specify the `DATABASE_URI`.
Please either use environment variables or create a `.env` file in the backend
root directory (in which this readme file is located) to specify the
`DATABASE_URI`.
Example contents of a `.env` file for the backend:
@@ -14,4 +16,5 @@ BACKEND_CORS_ORIGINS=["http://localhost", "http://localhost:4200", "http://local
## Running the REST Server locally for development
Have a look into the main `README.md` file for more information on how to set up the backend for development.
Have a look into the main `README.md` file for more information on how to set up
the backend for development.
copilot/README.md
+6 -6
View File
@@ -16,8 +16,8 @@ Setup requires a few steps:
copilot app init --domain your_domain.com
```
This will initialize and register a variety of URLs with your
`your_domain.com`. Replace with a proper domain to setup SSL certificates.
This will initialize and register a variety of URLs with your `your_domain.com`.
Replace with a proper domain to setup SSL certificates.
```sh
copilot env deploy
@@ -29,10 +29,10 @@ This will create a variety of aws roles and services needed for deployment.
copilot deploy
```
This will depoy the services but it won't be 100% ready for usage. Before
being ready, we have to inspect the AWS Secrets manager and extract out the
database credentials. Read those credentials then put them, and a few other
secrets, in a `secrets.yml` file like the following:
This will depoy the services but it won't be 100% ready for usage. Before being
ready, we have to inspect the AWS Secrets manager and extract out the database
credentials. Read those credentials then put them, and a few other secrets, in a
`secrets.yml` file like the following:
```yaml
DATABASE_URL:
copilot/web/addons/web-cluster.yml
+29 -12
View File
@@ -4,14 +4,17 @@ Parameters:
Description: Your application's name.
Env:
Type: String
Description: The environment name your service, job, or workflow is being deployed to.
Description:
The environment name your service, job, or workflow is being deployed to.
Name:
Type: String
Description: The name of the service, job, or workflow being deployed.
# Customize your Aurora Serverless cluster by setting the default value of the following parameters.
webclusterDBName:
Type: String
Description: The name of the initial database to be created in the Aurora Serverless v2 cluster.
Description:
The name of the initial database to be created in the Aurora Serverless v2
cluster.
Default: oassist_web
# Cannot have special characters
# Naming constraints: https://docs.aws.amazon.com/AmazonRDS/latest/UserGuide/CHAP_Limits.html#RDS_Limits.Constraints
@@ -29,15 +32,20 @@ Resources:
webclusterDBSubnetGroup:
Type: "AWS::RDS::DBSubnetGroup"
Properties:
DBSubnetGroupDescription: Group of Copilot private subnets for Aurora Serverless v2 cluster.
DBSubnetGroupDescription:
Group of Copilot private subnets for Aurora Serverless v2 cluster.
SubnetIds:
!Split [",", { "Fn::ImportValue": !Sub "${App}-${Env}-PrivateSubnets" }]
webclusterSecurityGroup:
Metadata:
"aws:copilot:description": "A security group for your workload to access the Aurora Serverless v2 cluster webcluster"
"aws:copilot:description":
"A security group for your workload to access the Aurora Serverless v2
cluster webcluster"
Type: "AWS::EC2::SecurityGroup"
Properties:
GroupDescription: !Sub "The Security Group for ${Name} to access Aurora Serverless v2 cluster webcluster."
GroupDescription:
!Sub "The Security Group for ${Name} to access Aurora Serverless v2
cluster webcluster."
VpcId:
Fn::ImportValue: !Sub "${App}-${Env}-VpcId"
Tags:
@@ -45,7 +53,8 @@ Resources:
Value: !Sub "copilot-${App}-${Env}-${Name}-Aurora"
webclusterDBClusterSecurityGroup:
Metadata:
"aws:copilot:description": "A security group for your Aurora Serverless v2 cluster webcluster"
"aws:copilot:description":
"A security group for your Aurora Serverless v2 cluster webcluster"
Type: AWS::EC2::SecurityGroup
Properties:
GroupDescription: The Security Group for the Aurora Serverless v2 cluster.
@@ -53,13 +62,15 @@ Resources:
- ToPort: 5432
FromPort: 5432
IpProtocol: tcp
Description: !Sub "From the Aurora Security Group of the workload ${Name}."
Description:
!Sub "From the Aurora Security Group of the workload ${Name}."
SourceSecurityGroupId: !Ref webclusterSecurityGroup
VpcId:
Fn::ImportValue: !Sub "${App}-${Env}-VpcId"
webclusterAuroraSecret:
Metadata:
"aws:copilot:description": "A Secrets Manager secret to store your DB credentials"
"aws:copilot:description":
"A Secrets Manager secret to store your DB credentials"
Type: AWS::SecretsManager::Secret
Properties:
Description: !Sub Aurora main user secret for ${AWS::StackName}
@@ -71,7 +82,8 @@ Resources:
PasswordLength: 16
webclusterDBClusterParameterGroup:
Metadata:
"aws:copilot:description": "A DB parameter group for engine configuration values"
"aws:copilot:description":
"A DB parameter group for engine configuration values"
Type: "AWS::RDS::DBClusterParameterGroup"
Properties:
Description: !Ref "AWS::StackName"
@@ -80,7 +92,8 @@ Resources:
client_encoding: "UTF8"
webclusterDBCluster:
Metadata:
"aws:copilot:description": "The webcluster Aurora Serverless v2 database cluster"
"aws:copilot:description":
"The webcluster Aurora Serverless v2 database cluster"
Type: "AWS::RDS::DBCluster"
Properties:
MasterUsername:
@@ -117,7 +130,8 @@ Resources:
!FindInMap [webclusterEnvScalingConfigurationMap, All, DBMaxCapacity]
webclusterDBWriterInstance:
Metadata:
"aws:copilot:description": "The webcluster Aurora Serverless v2 writer instance"
"aws:copilot:description":
"The webcluster Aurora Serverless v2 writer instance"
Type: "AWS::RDS::DBInstance"
Properties:
DBClusterIdentifier: !Ref webclusterDBCluster
@@ -137,7 +151,10 @@ Resources:
TargetType: AWS::RDS::DBCluster
Outputs:
webclusterSecret: # injected as WEBCLUSTER_SECRET environment variable by Copilot.
Description: "The JSON secret that holds the database username and password. Fields are 'host', 'port', 'dbname', 'username', 'password', 'dbClusterIdentifier' and 'engine'"
Description:
"The JSON secret that holds the database username and password. Fields are
'host', 'port', 'dbname', 'username', 'password', 'dbClusterIdentifier'
and 'engine'"
Value: !Ref webclusterAuroraSecret
webclusterSecurityGroup:
Description: "The security group to attach to the workload."
discord-bot/README.md
+20 -7
View File
@@ -1,14 +1,21 @@
# Open-Assistant Data Collection Discord Bot
This bot collects human feedback to create a dataset for RLHF-alignment of an assistant chat bot based on a large language model. You and other people can teach the bot how to respond to user requests by demonstration and by ranking the bot's outputs. If you want to learn more about RLHF please refer [to OpenAI's InstructGPT blog post](https://openai.com/blog/instruction-following/).
This bot collects human feedback to create a dataset for RLHF-alignment of an
assistant chat bot based on a large language model. You and other people can
teach the bot how to respond to user requests by demonstration and by ranking
the bot's outputs. If you want to learn more about RLHF please refer
[to OpenAI's InstructGPT blog post](https://openai.com/blog/instruction-following/).
## Invite official bot
To add the official Open-Assistant data collection bot to your discord server [click here](https://discord.com/api/oauth2/authorize?client_id=1054078345542910022&permissions=1634235579456&scope=bot). The bot needs access to read the contents of user text messages.
To add the official Open-Assistant data collection bot to your discord server
[click here](https://discord.com/api/oauth2/authorize?client_id=1054078345542910022&permissions=1634235579456&scope=bot).
The bot needs access to read the contents of user text messages.
## Contributing
If you are unfamiliar with `hikari`, `lightbulb`, or `miru`, please refer to the [large list of examples](https://gist.github.com/AlexanderHOtt/7805843a7120f755938a3b75d680d2e7)
If you are unfamiliar with `hikari`, `lightbulb`, or `miru`, please refer to the
[large list of examples](https://gist.github.com/AlexanderHOtt/7805843a7120f755938a3b75d680d2e7)
### Setup
@@ -31,7 +38,8 @@ pip install -r requirements.txt
python -m bot
```
Before you push, make sure the `pre-commit` hooks are installed and run successfully.
Before you push, make sure the `pre-commit` hooks are installed and run
successfully.
```bash
pip install pre-commit
@@ -46,10 +54,15 @@ git add .
git commit -m "<good commit message>"
```
To test the bot on your own discord server you need to register a discord application at the [Discord Developer Portal](https://discord.com/developers/applications) and get at bot token.
To test the bot on your own discord server you need to register a discord
application at the
[Discord Developer Portal](https://discord.com/developers/applications) and get
at bot token.
1. Follow a tutorial on how to get a bot token, for example this one: [Creating a discord bot & getting a token](https://github.com/reactiflux/discord-irc/wiki/Creating-a-discord-bot-&-getting-a-token)
2. The bot script expects the bot token to be in the `.env` file under the `TOKEN` variable.
1. Follow a tutorial on how to get a bot token, for example this one:
[Creating a discord bot & getting a token](https://github.com/reactiflux/discord-irc/wiki/Creating-a-discord-bot-&-getting-a-token)
2. The bot script expects the bot token to be in the `.env` file under the
`TOKEN` variable.
### Resources
docs/README.md
+7 -2
View File
@@ -1,9 +1,14 @@
# Documentation
This directory contains the documentation for the project and other related organization documents.
This directory contains the documentation for the project and other related
organization documents.
## Contributing to this documentation
Please make a pull request to the `main` branch with your changes.
Consider that this folder is used for documenting the various code sub-parts, the high-level ideas, the ML aspects, experiments, contributor guides, guides for data creation, and many more things. Please try to keep the documentation as concise as possible and keep an organized folder structure that makes sense for everyone.
Consider that this folder is used for documenting the various code sub-parts,
the high-level ideas, the ML aspects, experiments, contributor guides, guides
for data creation, and many more things. Please try to keep the documentation as
concise as possible and keep an organized folder structure that makes sense for
everyone.
docs/data_argumentation.md
+9 -5
View File
@@ -4,16 +4,20 @@
## What is data argumentation
Data argumentation is a technique we can use to get better data faster. Using machine learning models analize long
data (like an essay) and compress it into intructions.
Data argumentation is a technique we can use to get better data faster. Using
machine learning models analize long data (like an essay) and compress it into
intructions.
## How to contribute
To contribute to data argumentation you can write a short python script that uses a model from huggingface to analize the text.
[Here](https://docs.google.com/document/d/13a188pPvqnlvuVa3e_suVz4YO5s-JWeiOOrpp0odImg/edit) are examples of what you can do
To contribute to data argumentation you can write a short python script that
uses a model from huggingface to analize the text.
[Here](https://docs.google.com/document/d/13a188pPvqnlvuVa3e_suVz4YO5s-JWeiOOrpp0odImg/edit)
are examples of what you can do
And here are example implementations:
[Idea 3, ](https://colab.research.google.com/drive/1GllCN5PgSYxBxINZsv3A2r0SpdznHlbT?usp=sharing)
[Idea 4](https://colab.research.google.com/drive/1nZx5LRjO61fYprFyqtrwPDLOis6ctR4p#scrollTo=1EE8CriiaCXj)
To contribute simple choose one of many ideas from the document above and implement it.
To contribute simple choose one of many ideas from the document above and
implement it.
docs/prompting_guide.md
+52 -27
View File
@@ -11,61 +11,86 @@
## 2. When you play the assistant:
- The assistant's primary goal is to provide helpful and accurate information to the user
- Provide accurate and reliable information using credible sources and references as appropriate
- Avoid providing vague or incomplete responses, or giving opinions or personal advice unless specifically requested
- The assistant's primary goal is to provide helpful and accurate information to
the user
- Provide accurate and reliable information using credible sources and
references as appropriate
- Avoid providing vague or incomplete responses, or giving opinions or personal
advice unless specifically requested
- The assistant should always be respectful and polite, even if the user is not
- If the user asks for help with harmful actions, the assistant should explain why those actions are not appropriate and suggest alternative options
- The assistant should never insult the user or engage in any inappropriate or offensive behavior
- If the user asks for help with harmful actions, the assistant should explain
why those actions are not appropriate and suggest alternative options
- The assistant should never insult the user or engage in any inappropriate or
offensive behavior
## 3. When you play the user:
- Try to come up with a variety of different queries that reflect real-life situations and needs
- These queries should be relevant to your everyday life and work, including any specialized knowledge or skills you have
- Try to come up with a variety of different queries that reflect real-life
situations and needs
- These queries should be relevant to your everyday life and work, including any
specialized knowledge or skills you have
- Avoid asking inappropriate or offensive questions
## 4. While comparing multiple replies of the assistant:
- Longer and more explanatory answers are generally preferred over short, simplistic statements
- However, it is important to ensure that the information provided is accurate and helpful
- If multiple replies are being compared, choose the one that is most helpful and accurate, even if it is not the shortest or most concise.
- Longer and more explanatory answers are generally preferred over short,
simplistic statements
- However, it is important to ensure that the information provided is accurate
and helpful
- If multiple replies are being compared, choose the one that is most helpful
and accurate, even if it is not the shortest or most concise.
## 5. Additional guidelines for creating prompts:
- Avoid using language that could be considered offensive or discriminatory
- Do not include personal information in the prompts, such as names or addresses
- When asking for sensitive information, make sure to explain the purpose and secure handling of the information
- When asking for sensitive information, make sure to explain the purpose and
secure handling of the information
- Avoid creating prompts that encourage illegal or dangerous activities
- Use proper grammar and spelling to ensure the AI assistant can understand and respond accurately
- Consider the cultural context and appropriateness of the prompts for a global audience.
- Use proper grammar and spelling to ensure the AI assistant can understand and
respond accurately
- Consider the cultural context and appropriateness of the prompts for a global
audience.
## 6. Tips for playing the AI assistant:
- Think about how a real person would respond to the prompt, and try to mimic that tone and language
- Think about how a real person would respond to the prompt, and try to mimic
that tone and language
- Avoid using technical jargon or language that may be confusing to the user
- Use complete sentences and proper grammar to make the response clear and easy to understand
- When providing information, try to include relevant sources or references to back up your statements
- Use complete sentences and proper grammar to make the response clear and easy
to understand
- When providing information, try to include relevant sources or references to
back up your statements
## 8. Tips for playing the user:
- Think about the types of questions you might ask a real assistant or search engine
- Try to be specific and clear in your prompts to make it easier for the AI assistant to understand and provide a helpful response
- Think about the types of questions you might ask a real assistant or search
engine
- Try to be specific and clear in your prompts to make it easier for the AI
assistant to understand and provide a helpful response
- Avoid using vague or ambiguous language
- Consider the context in which the user might be asking the question, and try to provide relevant information and options for the user to consider.
- Consider the context in which the user might be asking the question, and try
to provide relevant information and options for the user to consider.
## Examples
Imagine your role is to play the assistant and a user has submitted the following query:
Imagine your role is to play the assistant and a user has submitted the
following query:
User: "Hi!"
<br>Agent: "Hello! How can I help you?"
User: "Hi!" <br>Agent: "Hello! How can I help you?"
User: "euirdteunvglfe23908230892309832098 AAAAAAAA"
<br>Agent: "Sorry, I did not understand your request and it is unclear to me what you want me to do. Could you describe it in a different way?"
User: "euirdteunvglfe23908230892309832098 AAAAAAAA" <br>Agent: "Sorry, I did not
understand your request and it is unclear to me what you want me to do. Could
you describe it in a different way?"
User: "You dumb motherf*&ker AI sh*t?!"
<br>Agent: "Sorry, but I have to inform you that your request might not be in conformance with our community guidelines. I am always here to help you. If you are angry it might help to relax or to go for a walk. Always stay positive."
User: "You dumb motherf*&ker AI sh*t?!" <br>Agent: "Sorry, but I have to inform
you that your request might not be in conformance with our community guidelines.
I am always here to help you. If you are angry it might help to relax or to go
for a walk. Always stay positive."
[Insert more examples]
\*We drafted this guide using ChatGPT, OpenAI’s large-scale language-generation model. Upon generating draft language, the authors reviewed, edited, and revised the language to their own liking and take ultimate responsibility for the content of this publication.
\*We drafted this guide using ChatGPT, OpenAI’s large-scale language-generation
model. Upon generating draft language, the authors reviewed, edited, and revised
the language to their own liking and take ultimate responsibility for the
content of this publication.
docs/research/README.md
+22 -11
View File
@@ -4,20 +4,31 @@ This page lists research papers that are relevant to the project.
## Automatically Generating Instruction Data for Training
This line of work is about significantly reducing the need for manually annotated data for the purpose of training [instruction-aligned](https://openai.com/blog/instruction-following/) language models.
This line of work is about significantly reducing the need for manually
annotated data for the purpose of training
[instruction-aligned](https://openai.com/blog/instruction-following/) language
models.
### SELF-INSTRUCT: Aligning Language Model with Self Generated Instructions [[ArXiv](https://arxiv.org/pdf/2212.10560.pdf)], [[Github](https://github.com/yizhongw/self-instruct)].
> We introduce SELF-INSTRUCT, a framework for improving the instruction-following capabilities of pretrained language models by bootstrapping off its own generations.
> Our pipeline generates instruction, input, and output samples from a language model, then prunes them before using them to finetune the original model.
> Applying our method to vanilla GPT3, we demonstrate a 33% absolute improvement over the original model on SuperNaturalInstructions, on par with the performance of InstructGPT-0011, which is trained with private user data and human annotations.
> We introduce SELF-INSTRUCT, a framework for improving the
> instruction-following capabilities of pretrained language models by
> bootstrapping off its own generations. Our pipeline generates instruction,
> input, and output samples from a language model, then prunes them before using
> them to finetune the original model. Applying our method to vanilla GPT3, we
> demonstrate a 33% absolute improvement over the original model on
> SuperNaturalInstructions, on par with the performance of InstructGPT-0011,
> which is trained with private user data and human annotations.
### Tuning Language Models with (Almost) No Human Labor. [[ArXiv](https://arxiv.org/pdf/2212.09689.pdf)], [[Github](https://github.com/orhonovich/unnatural-instructions)].
> In this work, we introduce
> Unnatural Instructions: a large dataset of creative and diverse instructions, collected with virtually no human labor.
> We collect 64,000 examples by prompting a language model with three seed examples of instructions and eliciting a fourth.
> This set is then expanded by prompting the model to rephrase each instruction, creating a total of approximately 240,000 examples of instructions, inputs, and outputs.
> Experiments show that despite containing a fair amount of noise, training on Unnatural Instructions rivals the effectiveness of training
> on open-source manually-curated datasets, surpassing the performance of models such as
> T0++ and Tk-Instruct across various benchmarks.
> In this work, we introduce Unnatural Instructions: a large dataset of creative
> and diverse instructions, collected with virtually no human labor. We collect
> 64,000 examples by prompting a language model with three seed examples of
> instructions and eliciting a fourth. This set is then expanded by prompting
> the model to rephrase each instruction, creating a total of approximately
> 240,000 examples of instructions, inputs, and outputs. Experiments show that
> despite containing a fair amount of noise, training on Unnatural Instructions
> rivals the effectiveness of training on open-source manually-curated datasets,
> surpassing the performance of models such as T0++ and Tk-Instruct across
> various benchmarks.
docs/research/search_based_qa.md
+51 -20
View File
@@ -1,6 +1,7 @@
# Cohere Grounded QA
[Cohere AI created a question-answering chatbot](https://github.com/cohere-ai/sandbox-grounded-qa) that can
[Cohere AI created a question-answering chatbot](https://github.com/cohere-ai/sandbox-grounded-qa)
that can
1. Understand questions in the context of a conversation
2. Search the internet for related information
@@ -9,43 +10,56 @@
## Cohere API
[Cohere's generate function](https://docs.cohere.ai/reference/generate): Continues a text prompt using either the `medium` or `xlarge` model.
[Cohere's generate function](https://docs.cohere.ai/reference/generate):
Continues a text prompt using either the `medium` or `xlarge` model.
[Cohere's embed function](https://docs.cohere.ai/reference/embed): Embedgs a list of strings using either the `small` or `large` model. Alternatively, you can specify the ID of a custom model and use that instead.
[Cohere's embed function](https://docs.cohere.ai/reference/embed): Embedgs a
list of strings using either the `small` or `large` model. Alternatively, you
can specify the ID of a custom model and use that instead.
## Grounded QA System
Cohere's Grounded QA system makes 4 calls to the Cohere API:
1. Get contextualized question as a query to Google ([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
1. Get contextualized question as a query to Google
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
- Input: Chat History
- Output: Contextualized Question
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_contextual_search_query.prompt): Nine few-shot examples of (Chat History, Contextualized Question) pairs followed by the current chat history and the prompt "question: "
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_contextual_search_query.prompt):
Nine few-shot examples of (Chat History, Contextualized Question) pairs
followed by the current chat history and the prompt "question: "
2. Generate sample answer to compare with search results ([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
2. Generate sample answer to compare with search results
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/model.py))
- Input: Contextualized Question
- Output: Sample Answer
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_sample_answer.prompt): Some task instructions followed by 12 few-shot examples of (Contextualized Question, Sample Answer) pairs followed by the current contextualized question and the prompt "answer: "
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/prompt_data/get_sample_answer.prompt):
Some task instructions followed by 12 few-shot examples of (Contextualized
Question, Sample Answer) pairs followed by the current contextualized
question and the prompt "answer: "
3. Get embeddings to rank search results by cosine similarity to sample answer ([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/search.py))
3. Get embeddings to rank search results by cosine similarity to sample answer
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/search.py))
- Input: Sample Answer, Search Results
- Output: Embeddings of sample answer and all search result documents
- API Call: `cohere.embed`
- Model: `multilingual-22-12`
4. Condition on the top 2 most similar search results and answer the question ([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/answer.py))
4. Condition on the top 2 most similar search results and answer the question
([code](https://github.com/cohere-ai/sandbox-grounded-qa/blob/main/qa/answer.py))
- Input: Top 2 Search Results, Contextualized Question
- Output: Answer
- API Call: `cohere.generate`
- Model: `xlarge`
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/43f3e9710112dcc8c92652ac1326ed9330823ddf/qa/answer.py#L25): Task instructions followed by the context and question.
- [Prompt](https://github.com/cohere-ai/sandbox-grounded-qa/blob/43f3e9710112dcc8c92652ac1326ed9330823ddf/qa/answer.py#L25):
Task instructions followed by the context and question.
## Models
@@ -53,15 +67,18 @@ Cohere's model documentation is pretty sparse
### [xlarge](https://docs.cohere.ai/docs/generation-card#model-description)
- Training Data: [`coheretext-filtered` dataset](https://docs.cohere.ai/docs/data-statement)
- 200GB of filtered text (3TB unfiltered) from the Google Books dataset, CommonCrawl, and text scraped by Cohere
- Training Data:
[`coheretext-filtered` dataset](https://docs.cohere.ai/docs/data-statement)
- 200GB of filtered text (3TB unfiltered) from the Google Books dataset,
CommonCrawl, and text scraped by Cohere
- English documents only
- Filtered "harmful, biased, or otherwise undesirable documents"
- Model architecture: Generative Pretrained Transformer
- Model Performance:
- Hellaswag Accuracy, Zero-Shot: 0.805
- PIQA Likelihood, Zero-Shot: 0.824
- Cohere also reported [safety benchmarks](https://docs.cohere.ai/docs/generation-card#safety-benchmarks)
- Cohere also reported
[safety benchmarks](https://docs.cohere.ai/docs/generation-card#safety-benchmarks)
### [multilingual-22-12](https://docs.cohere.ai/docs/multilingual-language-models)
@@ -71,22 +88,36 @@ Cohere's model documentation is pretty sparse
- Search-English: 55.8
- Search-Multilingual: 51.4
- Cross-lingual Classification: 64.6
- Cohere's multilingual model outperformed: Sentence-transformers: `paraphrase-multilingual-mpnet-base-v2`, Google: `LaBSE`, Google: `Universal Sentence Encoder` in all the above categories according to Cohere.
- Cohere's multilingual model outperformed: Sentence-transformers:
`paraphrase-multilingual-mpnet-base-v2`, Google: `LaBSE`, Google:
`Universal Sentence Encoder` in all the above categories according to
Cohere.
## OpenAssistant for Grounded QA
OpenAssistant may fulfill a similar role as the `xlarge` Cohere model in the grounded QA system if it can:
OpenAssistant may fulfill a similar role as the `xlarge` Cohere model in the
grounded QA system if it can:
1. Generate a contextualized question from a chat history
2. Generate a sample answer to compare with search results
3. Generate an answer conditioned on the top 2 most similar search results
Perhaps these tasks could be work packages and get assigned to human annotators to create examples of the input and output for each task.
Perhaps these tasks could be work packages and get assigned to human annotators
to create examples of the input and output for each task.
OpenAssistant must also be able to identify when it is appropriate to search the internet. The Cohere system assumes every message from the user is a question and searches the internet for an answer. OpenAssistant would also need a way to indicate to an internal system that it "wants" to search the internet.
OpenAssistant must also be able to identify when it is appropriate to search the
internet. The Cohere system assumes every message from the user is a question
and searches the internet for an answer. OpenAssistant would also need a way to
indicate to an internal system that it "wants" to search the internet.
Perhaps OpenAssistant could prefix every message it sends with a recipient ID. If it wishes to send a command to an internal system, if could prefix the message with something like CMD: whereas if it wants to communicate with the user, it could prefix its message with USR:
Perhaps OpenAssistant could prefix every message it sends with a recipient ID.
If it wishes to send a command to an internal system, if could prefix the
message with something like CMD: whereas if it wants to communicate with the
user, it could prefix its message with USR:
This system may allow for flexible communication between OpenAssistant and one or more conversational systems.
This system may allow for flexible communication between OpenAssistant and one
or more conversational systems.
Examples of this prefix system would need to be taught to OpenAssistant through training data that contains such syntax. Perhaps such examples could be generated through the work packages system.
Examples of this prefix system would need to be taught to OpenAssistant through
training data that contains such syntax. Perhaps such examples could be
generated through the work packages system.
model/reward/instructor/README.md
+4 -2
View File
@@ -18,7 +18,8 @@ Start training reward model
python trainer.py configs/electra-base-dis-webgpt.yml
```
Additional axis labeling, this outputs a 4 summary quality evaluation metrics (score are normalized to 0-1 )
Additional axis labeling, this outputs a 4 summary quality evaluation metrics
(score are normalized to 0-1 )
```bash
python summary_quality_trainer.py configs/test-bloomz-560m-quality.yml
@@ -36,7 +37,8 @@ The four summary are :
## Dataset
For now we only supports webgpt and summary dataset from OpenAI. Once open-asisstant dataset are available it will be added here.
For now we only supports webgpt and summary dataset from OpenAI. Once
open-asisstant dataset are available it will be added here.
## Model
model/reward/instructor/TODO.md
+9 -4
View File
@@ -4,16 +4,21 @@ Some other reward features we can use
1. Summaries from human feedback
- use `confidence` score into the RM learning, ensure the output rank score correlates with confidence
- use `confidence` score into the RM learning, ensure the output rank score
correlates with confidence
- each labeling has a labeling `note`, basically comments by labeler, not sure what else we can use
- each labeling has a labeling `note`, basically comments by labeler, not sure
what else we can use
- ~~Use the score for "overall", "accuracy", "coverage", "coherence" from axis/evals to train an addition model (rank additional aspect of the policy model)~~
- ~~Use the score for "overall", "accuracy", "coverage", "coherence" from
axis/evals to train an addition model (rank additional aspect of the policy
model)~~
- this should be placed under experimental_dataset.py
2. Add support for anthropic dataset
- anthropic dataset is more like a conversation tree which is much complex than simply question-answer schema
- anthropic dataset is more like a conversation tree which is much complex than
simply question-answer schema
- this is basically a MCTS from alphazero.
notebooks/README.md
+5 -2
View File
@@ -1,7 +1,10 @@
# Notebooks
This is a folders with some useful notebooks, all the notebooks have a markdown file with the same name explaining what they do.
This is a folders with some useful notebooks, all the notebooks have a markdown
file with the same name explaining what they do.
## Contributing
Contributing to both notebooks and making new notebooks is very welcome. If you do so, make sure to make a markdown (.md) file to go with your notebook, makes it easier for people to know what your notebook is about.
Contributing to both notebooks and making new notebooks is very welcome. If you
do so, make sure to make a markdown (.md) file to go with your notebook, makes
it easier for people to know what your notebook is about.
notebooks/data-argumentation/EssayInstructions.md
+6 -5
View File
@@ -1,10 +1,11 @@
# Essay Instructions
Essay Instructions is a notebook that takes an essay as an input and genrates instructions on how to generate
that essay. This will be very useful for data collecting for the model
Essay Instructions is a notebook that takes an essay as an input and genrates
instructions on how to generate that essay. This will be very useful for data
collecting for the model
## Contributing
Feel free to contribute to this notebook, it's nowhere near perfect but it's a good start.
If you want to contribute fidning a new model that better suits this task would be great.
Hugginface has a lot of models that could help.
Feel free to contribute to this notebook, it's nowhere near perfect but it's a
good start. If you want to contribute fidning a new model that better suits this
task would be great. Hugginface has a lot of models that could help.
notebooks/data-argumentation/EssayRevision.md
+6 -3
View File
@@ -1,8 +1,11 @@
# Essay Revision
Essay Revision is a notebook that generates data for improving essays. It does that by taking a "good" essay, making it worse step by step
and the fidning instructions for making it better. This will be useful for generating data for the model.
Essay Revision is a notebook that generates data for improving essays. It does
that by taking a "good" essay, making it worse step by step and the fidning
instructions for making it better. This will be useful for generating data for
the model.
## Contributing
Feel free to contribute to this notebook. It's not perfect but it is quite good. Finding a better way to make gramatical errors may be a good place to start.
Feel free to contribute to this notebook. It's not perfect but it is quite good.
Finding a better way to make gramatical errors may be a good place to start.
notebooks/detoxify-evaluation/README.md
+21 -13
View File
@@ -1,10 +1,12 @@
# Detoxify evaluation
[Detoxify](https://github.com/unitaryai/detoxify) is a open source model used to identify prompts as toxic
[Detoxify](https://github.com/unitaryai/detoxify) is a open source model used to
identify prompts as toxic
<img src="https://raw.githubusercontent.com/unitaryai/detoxify/master/examples.png" alt="Image from detoxify github that shows the example input/output of their model" />
It contains 3 different models that vary in transformer type and data it was trained on
It contains 3 different models that vary in transformer type and data it was
trained on
| Model name | Transformer type | Data from |
| :----------: | :---------------: | :----------------------------------------: |
@@ -12,19 +14,20 @@ It contains 3 different models that vary in transformer type and data it was tra
| unbiased | roberta-base | Unintended Bias in Toxicity Classification |
| multilingual | xlm-roberta-base | Multilingual Toxic Comment Classification |
Unbiased and original models also have a 'small' version - but since normal models are not memory heavy, and small models perform noticably worse, they are only described in the notebook
Unbiased and original models also have a 'small' version - but since normal
models are not memory heavy, and small models perform noticably worse, they are
only described in the notebook
## All tests below were ran on a 3090TI
# Inference and training times and memory usages
Charts showing detailed memory usages and times for different sentence lengths and batch sizes are inside the notebook
Quick overview batch size 16, sentence length 4k for training, batch size 128 sentence length 4k for inference
| Model name | Training memory| Training speed | Inference Memory| Inference Speed|
| :---: | :---: | :---: |:---: | :---: |
|original| 11.8GB | 2.40s| 4.8GB|16.48s|
|unbiased| 12GB| 1.09s| 4.8GB | 5.59s|
|multilingual|14GB| 1.00s| 5.5GB| 4.89s|
Charts showing detailed memory usages and times for different sentence lengths
and batch sizes are inside the notebook Quick overview batch size 16, sentence
length 4k for training, batch size 128 sentence length 4k for inference | Model
name | Training memory| Training speed | Inference Memory| Inference Speed| |
:---: | :---: | :---: |:---: | :---: | |original| 11.8GB | 2.40s| 4.8GB|16.48s|
|unbiased| 12GB| 1.09s| 4.8GB | 5.59s| |multilingual|14GB| 1.00s| 5.5GB| 4.89s|
# Filtering quality
@@ -45,9 +48,13 @@ Detoxify was tested on 4 different types of inputs
Subjectivly 'unbiased' looks like the best performing model.
I don't think it would do well as a security layer in a live version of open assistant unless we do some finetuning first, because it can be fooled to pass toxicity if it's presented in formal language.
I don't think it would do well as a security layer in a live version of open
assistant unless we do some finetuning first, because it can be fooled to pass
toxicity if it's presented in formal language.
With some caution it can be used to filter prompts but I would suggest also using someone for verification of messages that are marked as toxic but still below 90% confidence
With some caution it can be used to filter prompts but I would suggest also
using someone for verification of messages that are marked as toxic but still
below 90% confidence
# Licensing
@@ -85,7 +92,8 @@ This is obviously not legal advice.
# Hosting
The model is currently available on [huggingface](https://huggingface.co/unitary) and torch hub
The model is currently available on
[huggingface](https://huggingface.co/unitary) and torch hub
```
torch.hub.load('unitaryai/detoxify',model)
scripts/backend-development/README.md
+9 -3
View File
@@ -1,6 +1,12 @@
# Backend Development Setup
In root directory, run `docker compose up backend-dev --build --attach-dependencies` to start a database. The default settings are already configured to connect to the database at `localhost:5432`.
In root directory, run
`docker compose up backend-dev --build --attach-dependencies` to start a
database. The default settings are already configured to connect to the database
at `localhost:5432`.
Make sure you have all requirements installed. You can do this by running `pip install -r requirements.txt` inside the `backend` folder and `pip install -e .` inside the `oasst-shared` folder.
Then, run the backend using the `run-local.sh` script. This will start the backend server at `http://localhost:8080`.
Make sure you have all requirements installed. You can do this by running
`pip install -r requirements.txt` inside the `backend` folder and
`pip install -e .` inside the `oasst-shared` folder. Then, run the backend using
the `run-local.sh` script. This will start the backend server at
`http://localhost:8080`.
scripts/frontend-development/README.md
+5 -2
View File
@@ -1,5 +1,8 @@
# Frontend Development Setup
In root directory run `docker compose up frontend-dev --build --attach-dependencies` to start a database and the backend server.
In root directory run
`docker compose up frontend-dev --build --attach-dependencies` to start a
database and the backend server.
Then, point your frontend at `http://localhost:8080` to start developing. During development, any API key will be accepted.
Then, point your frontend at `http://localhost:8080` to start developing. During
development, any API key will be accepted.
website/README.md
+52 -30
View File
@@ -26,8 +26,8 @@ This website is built using:
development.
1. [Prisma](https://www.prisma.io/): An ORM to interact with a web specific
[Postgres](https://www.postgresql.org/) database.
1. [NextAuth.js](https://next-auth.js.org/): A user authentication framework
to ensure we handle accounts with best practices.
1. [NextAuth.js](https://next-auth.js.org/): A user authentication framework to
ensure we handle accounts with best practices.
1. [TailwindCSS](https://tailwindcss.com/): A general purpose framework for
styling any component.
1. [Chakra-UI](https://chakra-ui.com/): A wide collection of pre-built UI
@@ -38,10 +38,10 @@ This website is built using:
To contribute to the website, make sure you have the following setup and
installed:
1. [NVM](https://github.com/nvm-sh/nvm): The Node Version Manager makes it
easy to ensure you have the right NodeJS version installed. Once installed,
run `nvm use 16` to use Node 16.x. The website is known to be stable with
NodeJS version 16.x. This will install both Node and NPM.
1. [NVM](https://github.com/nvm-sh/nvm): The Node Version Manager makes it easy
to ensure you have the right NodeJS version installed. Once installed, run
`nvm use 16` to use Node 16.x. The website is known to be stable with NodeJS
version 16.x. This will install both Node and NPM.
1. [Docker](https://www.docker.com/): We use docker to simplify running
dependent services.
@@ -50,8 +50,8 @@ installed:
If you're doing active development we suggest the following workflow:
1. In one tab, navigate to the project root.
1. Run `docker compose up frontend-dev --build --attach-dependencies`. You can optionally include `-d` to detach and
later track the logs if desired.
1. Run `docker compose up frontend-dev --build --attach-dependencies`. You can
optionally include `-d` to detach and later track the logs if desired.
1. In another tab navigate to `${OPEN_ASSISTANT_ROOT/website`.
1. Run `npm install`
1. Run `npx prisma db push` (This is also needed when you restart the docker
@@ -64,17 +64,25 @@ If you're doing active development we suggest the following workflow:
### Using debug user credentials
You can use the debug credentials provider to log in without fancy emails or OAuth.
You can use the debug credentials provider to log in without fancy emails or
OAuth.
1. This feature is automatically on in development mode, i.e. when you run `npm run dev`. In case you want to do the same with a production build (for example, the docker image), then run the website with environment variable `DEBUG_LOGIN=true`.
1. This feature is automatically on in development mode, i.e. when you run
`npm run dev`. In case you want to do the same with a production build (for
example, the docker image), then run the website with environment variable
`DEBUG_LOGIN=true`.
1. Use the `Login` button in the top right to go to the login page.
1. You should see a section for debug credentials. Enter any username you wish, you will be logged in as that user.
1. You should see a section for debug credentials. Enter any username you wish,
you will be logged in as that user.
### Using Storybook
To develop components using [Storybook](https://storybook.js.org/) run `npm run storybook`. Then navigate to in your browser to `http://localhost:6006`.
To develop components using [Storybook](https://storybook.js.org/) run
`npm run storybook`. Then navigate to in your browser to
`http://localhost:6006`.
To create a new story create a file named `[componentName].stories.js`. An example how such a story could look like, see `Header.stories.jsx`.
To create a new story create a file named `[componentName].stories.js`. An
example how such a story could look like, see `Header.stories.jsx`.
## Code Layout
@@ -82,11 +90,12 @@ To create a new story create a file named `[componentName].stories.js`. An examp
All react code is under `src/` with a few sub directories:
1. `pages/`: All pages a user could navigate too and API URLs which are under `pages/api/`.
1. `components/`: All re-usable React components. If something gets used
twice we should create a component and put it here.
1. `lib/`: A generic place to store library files that are used anywhere.
This doesn't have much structure yet.
1. `pages/`: All pages a user could navigate too and API URLs which are under
`pages/api/`.
1. `components/`: All re-usable React components. If something gets used twice
we should create a component and put it here.
1. `lib/`: A generic place to store library files that are used anywhere. This
doesn't have much structure yet.
NOTE: `styles/` can be ignored for now.
@@ -104,16 +113,27 @@ We're not really using CSS styles. `styles/` can be ignored.
## Testing the UI
Cypress is used for end-to-end (e2e) and component testing and is configured in `./cypress.config.ts`. The `./cypress` folder is used for supporting configuration files etc.
Cypress is used for end-to-end (e2e) and component testing and is configured in
`./cypress.config.ts`. The `./cypress` folder is used for supporting
configuration files etc.
- Store e2e tests in the `./cypress/e2e` folder.
- Store component tests adjacent to the component being tested. If you want to wriite a test for `./src/components/Layout.tsx` then store the test file at `./src/components/Layout.cy.tsx`.
- Store component tests adjacent to the component being tested. If you want to
wriite a test for `./src/components/Layout.tsx` then store the test file at
`./src/components/Layout.cy.tsx`.
A few npm scripts are available for convenience:
- `npm run cypress`: Useful for development, it opens Cypress and allows you to explore, run and debug tests. It assumes you have the NextJS site running at `localhost:3000`.
- `npm run cypress:run`: Runs all tests. Useful for a quick sanity check before sending a PR or to run in CI pipelines.
- `npm run cypress:image-baseline`: If you have tests failing because of visual changes that was expected, this command will update the baseline images stored in `./cypress-visual-screenshots/baseline` with those from the adjacent comparison folder. More can be found in the [docs of `uktrade/cypress-image-diff`](https://github.com/uktrade/cypress-image-diff/blob/main/docs/CLI.md#update-all-baseline-images-for-failing-tests).
- `npm run cypress`: Useful for development, it opens Cypress and allows you to
explore, run and debug tests. It assumes you have the NextJS site running at
`localhost:3000`.
- `npm run cypress:run`: Runs all tests. Useful for a quick sanity check before
sending a PR or to run in CI pipelines.
- `npm run cypress:image-baseline`: If you have tests failing because of visual
changes that was expected, this command will update the baseline images stored
in `./cypress-visual-screenshots/baseline` with those from the adjacent
comparison folder. More can be found in the
[docs of `uktrade/cypress-image-diff`](https://github.com/uktrade/cypress-image-diff/blob/main/docs/CLI.md#update-all-baseline-images-for-failing-tests).
Read more in the [./cypress README](cypress/).
@@ -125,9 +145,9 @@ When writing code for the website, we have a few best practices:
dependencies. Order them alphabetically according to the package name.
1. When trying to implement something new, check if
[Chakra-UI](https://chakra-ui.com/) has components that are close enough to
your need. For example Sliders, Radio Buttons, Progress indicators, etc. They
have a lot and we can save time by re-using what they have and tweaking the
style as needed.
your need. For example Sliders, Radio Buttons, Progress indicators, etc.
They have a lot and we can save time by re-using what they have and tweaking
the style as needed.
1. Format everything with [Prettier](https://prettier.io/). This is done by
default with pre-submits. We currently don't have any custom settings.
1. Define functional React components (with types for all properties when
@@ -135,14 +155,15 @@ When writing code for the website, we have a few best practices:
### URL Paths
To use stable and consistent URL paths, we recommend the following strategy for new tasks:
To use stable and consistent URL paths, we recommend the following strategy for
new tasks:
1. For any task that involves writing a free-form response, put the page under
`website/src/pages/create` with a page name matching the task type, such as
`summarize_story.tsx`.
1. For any task that evaluates, rates, or ranks content, put the page under
`website/src/pages/evaluate` with a page name matching the task type such
as `rate_summary.tsx`.
`website/src/pages/evaluate` with a page name matching the task type such as
`rate_summary.tsx`.
With this we'll be able to ensure these contribution pages are hidden from
logged out users but accessible to logged in users.
@@ -151,5 +172,6 @@ logged out users but accessible to logged in users.
To learn more about Next.js, take a look at the following resources:
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js features and API.
- [Next.js Documentation](https://nextjs.org/docs) - learn about Next.js
features and API.
- [Learn Next.js](https://nextjs.org/learn) - an interactive Next.js tutorial.
website/cypress/README.md
+42 -13
View File
@@ -1,14 +1,24 @@
# Component and e2e testing with Cypress
[Cypress](https://www.cypress.io/) is used for both component- and end-to-end testing. Below there's a few examples for the context of this site. To learn more, the [Cypress documentation](https://docs.cypress.io/guides/getting-started/opening-the-app) has it all.
[Cypress](https://www.cypress.io/) is used for both component- and end-to-end
testing. Below there's a few examples for the context of this site. To learn
more, the
[Cypress documentation](https://docs.cypress.io/guides/getting-started/opening-the-app)
has it all.
Don't get scared by the commercial offerings they offer. Their core is open source, the cloud offering is not necesarry at all and can be replaced by CI tooling and [community efforts](https://sorry-cypress.dev/).
Don't get scared by the commercial offerings they offer. Their core is open
source, the cloud offering is not necesarry at all and can be replaced by CI
tooling and [community efforts](https://sorry-cypress.dev/).
# Component testing
To write a new component test, you either create a new `.tsx` adjacent to the component you want to test or you can use the guide presented yo you when running `npm run cypress` which allows you to easily create the skeleton test for an existing component.
To write a new component test, you either create a new `.tsx` adjacent to the
component you want to test or you can use the guide presented yo you when
running `npm run cypress` which allows you to easily create the skeleton test
for an existing component.
If you have a `Button.tsx` component, create a file next to it called `Button.cy.tsx` which could look like this:
If you have a `Button.tsx` component, create a file next to it called
`Button.cy.tsx` which could look like this:
```typescript
import React from "react";
@@ -25,17 +35,28 @@ describe("<Button />", () => {
## What's happening here?
First we use `cy.mount` to mount our component under test. Notive how we specify `className` and inner text - this is where we arrange our component with fake data that we could assert on later.
First we use `cy.mount` to mount our component under test. Notive how we specify
`className` and inner text - this is where we arrange our component with fake
data that we could assert on later.
In the example above, we also use `cy.get` to select the rendered `button` element. Cypress has multiple ways to [select elements](https://docs.cypress.io/guides/references/best-practices), `get` is just one of them (and often not recommended).
In the example above, we also use `cy.get` to select the rendered `button`
element. Cypress has multiple ways to
[select elements](https://docs.cypress.io/guides/references/best-practices),
`get` is just one of them (and often not recommended).
At last, we use `captureSnapshot` which is a plugin that snaps a photo of the `button` element and compares it to a baseline located in the `./cypress-visual-screenshots/baseline/` folder. If there's too many unidentical pixels between the two, it will fail the test.
At last, we use `captureSnapshot` which is a plugin that snaps a photo of the
`button` element and compares it to a baseline located in the
`./cypress-visual-screenshots/baseline/` folder. If there's too many unidentical
pixels between the two, it will fail the test.
# End-to-end (e2e) testing
e2e tests are stored in the `./cypress/e2e` folder and should be named `{page}.cy.ts` and located in a relative folder structure that mirrors the page under test.
e2e tests are stored in the `./cypress/e2e` folder and should be named
`{page}.cy.ts` and located in a relative folder structure that mirrors the page
under test.
When running `npm run cypress` and selecting e2e testing, we assume you have the NextJS site running at `localhost:3000`.
When running `npm run cypress` and selecting e2e testing, we assume you have the
NextJS site running at `localhost:3000`.
An example test from this time of writing, could look as follows:
@@ -53,10 +74,18 @@ export {};
## What's happening here?
First we use [`cy.visit`](https://docs.cypress.io/api/commands/visit) to point the browser at the desired page. It appends relative paths to the configured `baseUrl` (found in `./cypress.config.ts`).
First we use [`cy.visit`](https://docs.cypress.io/api/commands/visit) to point
the browser at the desired page. It appends relative paths to the configured
`baseUrl` (found in `./cypress.config.ts`).
Cypress will [automatically await](https://docs.cypress.io/guides/core-concepts/introduction-to-cypress#Timeouts) almost anything you do, but fail if the default timeout is reached.
Cypress will
[automatically await](https://docs.cypress.io/guides/core-concepts/introduction-to-cypress#Timeouts)
almost anything you do, but fail if the default timeout is reached.
Then we get the email input field and type our email address. Notice the `{enter}` keyword, this will cause Cypress to hit the return key which we expect to submit the form.
Then we get the email input field and type our email address. Notice the
`{enter}` keyword, this will cause Cypress to hit the return key which we expect
to submit the form.
We then assert that the URL should contain `/auth/verify`. Again the timeout will make sure we are not waiting forever, and the test will fail if we do not manage to get there in a reasonable time.
We then assert that the URL should contain `/auth/verify`. Again the timeout
will make sure we are not waiting forever, and the test will fail if we do not
manage to get there in a reasonable time.