Merging in main

.github/workflows/docker-build.yaml

@@ -0,0 +1,57 @@
+name: Build
+
+on:
+  workflow_call:
+    inputs:
+      folder:
+        required: true
+        type: string
+      image-name:
+        required: true
+        type: string
+      build-args:
+        required: false
+        type: string
+
+jobs:
+  build:
+    name: Build Images
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      packages: write
+    steps:
+      - uses: actions/checkout@v3
+      - name: Set up Docker Buildx
+        uses: docker/setup-buildx-action@v2.2.1
+      - name: Login to container registry
+        uses: docker/login-action@v2.1.0
+        with:
+          registry: ghcr.io
+          username: ${{ github.actor }}
+          password: ${{ secrets.GITHUB_TOKEN }}
+      - name: Get base registry
+        run: |
+          echo "REGISTRY=ghcr.io/${GITHUB_REPOSITORY,,}" >> $GITHUB_ENV
+      - name: Set tag prefix
+        if: github.ref_name != 'main'
+        run: |
+          echo "TAG_PREFIX=${{ github.ref_name }}-" >> $GITHUB_ENV
+      - name: Extract metadata (tags, labels) for Docker
+        id: meta
+        uses: docker/metadata-action@v4.1.1
+        with:
+          images: ${{ env.REGISTRY }}/${{ inputs.image-name }}
+          tags: |
+            type=sha,prefix=${{ env.TAG_PREFIX }},format=short
+            type=ref,event=tag
+      - name: Build and push Docker image
+        uses: docker/build-push-action@v3.2.0
+        with:
+          context: ${{ inputs.folder }}
+          build-args: ${{ inputs.build-args }}
+          push: true
+          tags: ${{ steps.meta.outputs.tags }}
+          labels: ${{ steps.meta.outputs.labels }}
+          cache-from: type=gha
+          cache-to: type=gha,mode=max

.github/workflows/release.yaml

@@ -0,0 +1,13 @@
+name: Release
+
+on:
+  release:
+    types: [released]
+
+jobs:
+  build-backend:
+    uses: ./.github/workflows/docker-build.yaml
+    with:
+      image-name: oasst-backend
+      folder: backend
+      build-args: ""

README.md

@@ -1,8 +1,8 @@
-# Open-Chat-GPT
+# Open-Assistant
 
-Open chat gpt is a project meant to give everyone access to a great chat based large language model.
+Open Assistant is a project meant to give everyone access to a great chat based large language model.
 
-We believe that by doing this we will create a revolution in innovation in language. In the same way that stable-diffusion helped the world make art and images in new ways we hope open chat gpt can help improve the world by improving language itself.
+We believe that by doing this we will create a revolution in innovation in language. In the same way that stable-diffusion helped the world make art and images in new ways we hope Open Assistant can help improve the world by improving language itself.
 
 ## How can you help?
 
@@ -14,19 +14,18 @@ All open source projects begins with people like you. Open source is the belief
 
 [Join the LAION Discord Server!](https://discord.gg/RQFtmAmk)
 
-[Visit the Notion](https://ykilcher.com/open-chat-gpt)
+[Visit the Notion](https://ykilcher.com/open-assistant)
 
 ## Developer Setup
 
 Work is organized in the [project board](https://github.com/orgs/LAION-AI/projects/3).
 
-### Python Backend
-
-For a local developer setup, look into the `backend` folder to pull up a local database and backend.
+- To get started with development, if you want to work on the backend, have a look at `scripts/backend-development/README.md`.
+- If you want to work on any frontend, have a look at `scripts/frontend-development/README.md` to make a backend available.
 
 There is also a minimal implementation of a frontend in the `text-frontend` folder.
 
-We are using Python 3.10
+We are using Python 3.10 for the backend.
 
 Check out the [High-Level Protocol Architecture](https://www.notion.so/High-Level-Protocol-Architecture-6f1fd3551da74213b560ead369f132dc)
 

backend/Dockerfile

@@ -0,0 +1,9 @@
+FROM tiangolo/uvicorn-gunicorn-fastapi:python3.10
+
+COPY ./requirements.txt /app/requirements.txt
+
+RUN pip install --no-cache-dir --upgrade -r /app/requirements.txt
+
+ENV PORT 8080
+
+COPY ./app /app

backend/README.md

@@ -1,4 +1,4 @@
-# Open-Chat-GPT REST Backend
+# Open-Assistant REST Backend
 
 ## REST Server Configuration
 
@@ -8,14 +8,10 @@ Example contents of a `.env` file for the backend:
 
 ```
 DATABASE_URI="postgresql://<username>:<password>@<host>/<database_name>"
-BACKEND_CORS_ORIGINS=["http://localhost", "http://localhost:4200", "http://localhost:3000", "http://localhost:8080", "https://localhost", "https://localhost:4200", "https://localhost:3000", "https://localhost:8080", "http://dev.ocgpt.laion.ai", "https://stag.ocgpt.laion.ai", "https://ocgpt.laion.ai"]
+BACKEND_CORS_ORIGINS=["http://localhost", "http://localhost:4200", "http://localhost:3000", "http://localhost:8080", "https://localhost", "https://localhost:4200", "https://localhost:3000", "https://localhost:8080", "http://dev.oasst.laion.ai", "https://stag.oasst.laion.ai", "https://oasst.laion.ai"]
 
 ```
 
 ## Running the REST Server locally for development
 
-First, install the requirements in `requirements.txt`.
-Then, run two terminals (note the working directory for each):
-
-- Terminal 1, to go `backend/scripts` and run `docker-compose up`. This will start postgres.
-- Terminal 2, to go `backend` and run `scripts/run-local.sh`. This will start the REST server.
+Have a look into the main `README.md` file for more information on how to set up the backend for development.

backend/app/alembic.ini

@@ -8,7 +8,7 @@ script_location = %(here)s/alembic
 # Uncomment the line below if you want the files to be prepended with date and time
 # see https://alembic.sqlalchemy.org/en/latest/tutorial.html#editing-the-ini-file
 # for all available tokens
-# file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
+file_template = %%(year)d_%%(month).2d_%%(day).2d_%%(hour).2d%%(minute).2d-%%(rev)s_%%(slug)s
 
 # sys.path path, will be prepended to sys.path if present.
 # defaults to the current working directory.

backend/app/alembic/env.py

@@ -3,7 +3,7 @@ from logging.config import fileConfig
 
 import sqlmodel
 from alembic import context
-from app import models  # noqa: F401
+from oasst import models  # noqa: F401
 from sqlalchemy import engine_from_config, pool
 
 # this is the Alembic Config object, which provides
@@ -68,6 +68,8 @@ def run_migrations_online() -> None:
         context.configure(connection=connection, target_metadata=target_metadata)
 
         with context.begin_transaction():
+            context.get_context()._ensure_version_table()
+            connection.execute("LOCK TABLE alembic_version IN ACCESS EXCLUSIVE MODE")
             context.run_migrations()
 
 

backend/app/alembic/versions/2022_12_17_2230-6368515778c5_add_auth_method_to_person.py

@@ -0,0 +1,30 @@
+# -*- coding: utf-8 -*-
+"""add auth_method to person
+
+Revision ID: 6368515778c5
+Revises: cd7de470586e
+Create Date: 2022-12-17 17:57:33.022549
+
+"""
+import sqlalchemy as sa
+from alembic import op
+
+# revision identifiers, used by Alembic.
+revision = "6368515778c5"
+down_revision = "cd7de470586e"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.add_column("person", sa.Column("auth_method", sa.String(length=128), nullable=True))
+    op.execute("UPDATE person SET auth_method = 'local'")
+    op.alter_column("person", "auth_method", nullable=False)
+    # ### end Alembic commands ###
+
+
+def downgrade() -> None:
+    # ### commands auto generated by Alembic - please adjust! ###
+    op.drop_column("person", "auth_method")
+    # ### end Alembic commands ###

backend/app/main.py

@@ -4,9 +4,9 @@ from pathlib import Path
 import alembic.command
 import alembic.config
 import fastapi
-from app.api.v1.api import api_router
-from app.config import settings
 from loguru import logger
+from oasst.api.v1.api import api_router
+from oasst.config import settings
 from starlette.middleware.cors import CORSMiddleware
 
 app = fastapi.FastAPI(title=settings.PROJECT_NAME, openapi_url=f"{settings.API_V1_STR}/openapi.json")
@@ -27,7 +27,7 @@ if settings.UPDATE_ALEMBIC:
     def alembic_upgrade():
         logger.info("Attempting to upgrade alembic on startup")
         try:
-            alembic_ini_path = Path(__file__).parent.parent / "alembic.ini"
+            alembic_ini_path = Path(__file__).parent / "alembic.ini"
             alembic_cfg = alembic.config.Config(str(alembic_ini_path))
             alembic_cfg.set_main_option("sqlalchemy.url", settings.DATABASE_URI)
             alembic.command.upgrade(alembic_cfg, "head")

backend/app/oasst/api/deps.py

@@ -3,12 +3,12 @@ from secrets import token_hex
 from typing import Generator
 from uuid import UUID
 
-from app.config import settings
-from app.database import engine
-from app.models import ApiClient
 from fastapi import HTTPException, Security
 from fastapi.security.api_key import APIKey, APIKeyHeader, APIKeyQuery
 from loguru import logger
+from oasst.config import settings
+from oasst.database import engine
+from oasst.models import ApiClient
 from sqlmodel import Session
 from starlette.status import HTTP_403_FORBIDDEN
 

backend/app/oasst/api/v1/api.py

@@ -1,6 +1,6 @@
 # -*- coding: utf-8 -*-
-from app.api.v1 import tasks
 from fastapi import APIRouter
+from oasst.api.v1 import tasks
 
 api_router = APIRouter()
 api_router.include_router(tasks.router, prefix="/tasks", tags=["tasks"])

backend/app/oasst/api/v1/tasks.py

@@ -3,13 +3,13 @@ import random
 from typing import Any
 from uuid import UUID
 
-from app.api import deps
-from app.models.db_payload import TaskPayload
-from app.prompt_repository import PromptRepository
-from app.schemas import protocol as protocol_schema
 from fastapi import APIRouter, Depends, HTTPException
 from fastapi.security.api_key import APIKey
 from loguru import logger
+from oasst.api import deps
+from oasst.models.db_payload import TaskPayload
+from oasst.prompt_repository import PromptRepository
+from oasst.schemas import protocol as protocol_schema
 from sqlmodel import Session
 from starlette.status import HTTP_400_BAD_REQUEST
 

backend/app/oasst/config.py

@@ -5,10 +5,11 @@ from pydantic import AnyHttpUrl, BaseSettings, PostgresDsn, validator
 
 
 class Settings(BaseSettings):
-    PROJECT_NAME: str = "open-chatGPT backend"
+    PROJECT_NAME: str = "open-assistant backend"
     API_V1_STR: str = "/api/v1"
 
-    POSTGRES_SERVER: str = "localhost:5432"
+    POSTGRES_HOST: str = "localhost"
+    POSTGRES_PORT: str = "5432"
     POSTGRES_USER: str = "postgres"
     POSTGRES_PASSWORD: str = "postgres"
     POSTGRES_DB: str = "postgres"
@@ -24,7 +25,8 @@ class Settings(BaseSettings):
             scheme="postgresql",
             user=values.get("POSTGRES_USER"),
             password=values.get("POSTGRES_PASSWORD"),
-            host=values.get("POSTGRES_SERVER"),
+            host=values.get("POSTGRES_HOST"),
+            port=values.get("POSTGRES_PORT"),
             path=f"/{values.get('POSTGRES_DB') or ''}",
         )
 

backend/app/oasst/database.py

@@ -1,5 +1,5 @@
 # -*- coding: utf-8 -*-
-from app.config import settings
+from oasst.config import settings
 from sqlmodel import create_engine
 
 if settings.DATABASE_URI is None:

backend/app/oasst/models/db_payload.py

@@ -1,8 +1,8 @@
 # -*- coding: utf-8 -*-
 from typing import Literal
 
-from app.models.payload_column_type import payload_type
-from app.schemas import protocol as protocol_schema
+from oasst.models.payload_column_type import payload_type
+from oasst.schemas import protocol as protocol_schema
 from pydantic import BaseModel
 
 

backend/app/oasst/models/person.py

@@ -18,6 +18,7 @@ class Person(SQLModel, table=True):
         ),
     )
     username: str = Field(nullable=False, max_length=128)
+    auth_method: str = Field(nullable=False, max_length=128, default="local")
     display_name: str = Field(nullable=False, max_length=256)
     created_date: Optional[datetime] = Field(
         sa_column=sa.Column(sa.DateTime(), nullable=False, server_default=sa.func.current_timestamp())

backend/app/oasst/prompt_repository.py

@@ -3,11 +3,11 @@ from datetime import datetime
 from typing import Optional
 from uuid import UUID, uuid4
 
-import app.models.db_payload as db_payload
-from app.models import ApiClient, Person, Post, PostReaction, WorkPackage
-from app.models.payload_column_type import PayloadContainer
-from app.schemas import protocol as protocol_schema
+import oasst.models.db_payload as db_payload
 from loguru import logger
+from oasst.models import ApiClient, Person, Post, PostReaction, WorkPackage
+from oasst.models.payload_column_type import PayloadContainer
+from oasst.schemas import protocol as protocol_schema
 from sqlmodel import Session
 
 
@@ -22,7 +22,13 @@ class PromptRepository:
         if not user:
             return None
         person: Person = (
-            self.db.query(Person).filter(Person.api_client_id == self.api_client.id, Person.username == user.id).first()
+            self.db.query(Person)
+            .filter(
+                Person.api_client_id == self.api_client.id,
+                Person.username == user.id,
+                Person.auth_method == user.auth_method,
+            )
+            .first()
         )
         if person is None:
             # user is unknown, create new record

backend/scripts/run-local.sh

@@ -1,5 +0,0 @@
-#!/usr/bin/env bash
-
-export ALLOW_ANY_API_KEY=True
-
-uvicorn app.main:app --reload

bot/Dockerfile

@@ -0,0 +1,7 @@
+FROM python:3.10-slim-bullseye
+RUN mkdir /app
+ADD requirements.txt /app/requirements.txt
+WORKDIR /app
+RUN pip install -r requirements.txt
+ADD . /app
+CMD ["python", "bot.py"]

bot/README.md

@@ -1,6 +1,6 @@
-# open-chat-gpt
+# open-assistant
 
-This is the github repo for the open-chat-gpt project.
+This is the github repo for the open-assistant project.
 We are currently building a discord bot in order to make everyone contribute with great prompts and answers.
 Join us!
 https://discord.gg/ZUfPw6jP

bot/bot.py

@@ -33,7 +33,7 @@ guild_ids = [TEST_GUILD, TEST_GUILD_LAION]
 
 
 # Initiate the client and command tree to create slash commands.
-class OpenChatGPTClient(discord.Client):
+class OpenAssistantClient(discord.Client):
     def __init__(self, *, intents: discord.Intents):
         super().__init__(intents=intents)
         self.tree = app_commands.CommandTree(self)
@@ -59,7 +59,7 @@ class OpenChatGPTClient(discord.Client):
 # List the set of intents needed for commands to operate properly.
 intents = discord.Intents.default()
 intents.message_content = True
-client = OpenChatGPTClient(intents=intents)
+client = OpenAssistantClient(intents=intents)
 
 
 class LikeButton(discord.ui.Button):

bot/setup.py

@@ -12,11 +12,11 @@ if __name__ == "__main__":
     REQUIREMENTS = _read_reqs("requirements.txt")
 
     setup(
-        name="open-chat-gpt",
+        name="open-assistant",
         packages=find_packages(),
         version="0.0.1",
         license="Apache 2.0",
-        description="A Discord Bot for collecting and ranking prompts to train an Open ChatGPT",
+        description="A Discord Bot for collecting and ranking prompts to train an Open Assistant",
         keywords=["machine learning", "natural language processing", "discord"],
         install_requires=REQUIREMENTS,
         classifiers=[

scripts/backend-development/README.md

@@ -0,0 +1,6 @@
+# Backend Development Setup
+
+Run `docker compose up` to start a database. The default settings are already configured to connect to the database at `localhost:5432`.
+
+Make sure you have all requirements installed. You can do this by running `pip install -r requirements.txt` inside the `backend` folder.
+Then, run the backend using the `run-local.sh` script. This will start the backend server at `http://localhost:8080`.

scripts/backend-development/run-local.sh

@@ -0,0 +1,11 @@
+#!/usr/bin/env bash
+parent_path=$( cd "$(dirname "${BASH_SOURCE[0]}")" ; pwd -P )
+
+# switch to backend directory
+pushd "$parent_path/../../backend/app"
+
+export ALLOW_ANY_API_KEY=True
+
+uvicorn main:app --reload --port 8080 --host 0.0.0.0
+
+popd

scripts/frontend-development/README.md

@@ -0,0 +1,5 @@
+# Frontend Development Setup
+
+Run `docker compose up --build` to start a database and the backend server.
+
+Then, point your frontend at `http://localhost:8080` to start developing. During development, any API key will be accepted.

scripts/frontend-development/docker-compose.yaml

@@ -0,0 +1,27 @@
+version: "3.7"
+
+services:
+  db:
+    extends:
+      file: ../backend-development/docker-compose.yaml
+      service: db
+    healthcheck:
+      test: ["CMD", "pg_isready", "-U", "postgres"]
+      interval: 2s
+      timeout: 2s
+      retries: 10
+  adminer:
+    extends:
+      file: ../backend-development/docker-compose.yaml
+      service: adminer
+  backend:
+    build: ../../backend/.
+    image: oasst-backend
+    environment:
+      - POSTGRES_HOST=db
+      - ALLOW_ANY_API_KEY=True
+    depends_on:
+      db:
+        condition: service_healthy
+    ports:
+      - "8080:8080"

text-frontend/__main__.py

@@ -25,7 +25,7 @@ def _render_message(message: dict) -> str:
 
 
 @app.command()
-def main(backend_url: str = "http://127.0.0.1:8000", api_key: str = "DUMMY_KEY"):
+def main(backend_url: str = "http://127.0.0.1:8080", api_key: str = "DUMMY_KEY"):
     """Simple REPL frontend."""
 
     def _post(path: str, json: dict) -> dict:

website/.env.example

@@ -1,4 +1,4 @@
-FASTAPI_URL=http://xamla.com:8000
+FASTAPI_URL=http://xamla.com:8080
 FASTAPI_KEY=magic_key
 
 DISCORD_CLIENT_ID=your_discord_bot_client_id

website/package.json

@@ -1,6 +1,6 @@
 {
   "name": "website",
-  "homepage": "http://projects.laion.ai.github.io/Open-Chat-GPT",
+  "homepage": "http://projects.laion.ai.github.io/Open-Assistant",
   "version": "0.1.0",
   "private": true,
   "scripts": {

website/pages/index.js

@@ -19,9 +19,9 @@ export default function Home() {
     return (
       <div className={styles.App}>
         <header className={styles.AppHeader}>
-          <h2>Open Chat Gpt</h2>
+          <h2>Open Assistant</h2>
           <p>
-            Open chat gpt is a project meant to give everyone access to a great
+            Open Assistant is a project meant to give everyone access to a great
             chat based large language model.
           </p>
 
@@ -29,7 +29,7 @@ export default function Home() {
           <p>
             We believe that by doing this we will create a revolution in
             innovation in language. In the same way that stable-diffusion helped
-            the world make art and images in new ways we hope open chat gpt can
+            the world make art and images in new ways we hope Open Assistant can
             help improve the world by improving language itself.
           </p>
 
@@ -59,9 +59,7 @@ export default function Home() {
   return (
     <div className={styles.App}>
       <header className={styles.AppHeader}>
-        {/* <img src={logo} className="App-logo" alt="logo" /> */}
-
-        <h2>Open Chat Gpt</h2>
+        <h2>Open Assistant</h2>
 
         <p>You are logged in</p>
 

website/public/index.html

@@ -5,7 +5,7 @@
     <link rel="icon" href="%PUBLIC_URL%/favicon.ico" />
     <meta name="viewport" content="width=device-width, initial-scale=1" />
     <meta name="theme-color" content="#000000" />
-    <meta name="description" content="Open-Chat-GPT" />
+    <meta name="description" content="Open-Assistant" />
     <link rel="apple-touch-icon" href="%PUBLIC_URL%/logo192.png" />
     <!--
       manifest.json provides metadata used when your web app is installed on a
@@ -21,7 +21,7 @@
       work correctly both with client-side routing and a non-root public URL.
       Learn how to configure a non-root public URL by running `npm run build`.
     -->
-    <title>Open Chat GPT</title>
+    <title>Open Assistant</title>
   </head>
   <body>
     <noscript>You need to enable JavaScript to run this app.</noscript>