Commit Graph
49 Commits
Author SHA1 Message Date
Andreas KöpfandGitHub f3ffde47ff add preferred lonely_children extension (#942)
* add preferred lonely_children extension

* simplify sampling process, lower the probability to 25%

* exclude parents for replies that were recently used

* lonely children := count > 0

* consider only tasks not done for parent exclusion

* increase lonely child sampling probability
2023-01-26 23:00:54 +01:00
Andreas KöpfandGitHub 558b207013 Add /messages/{message_id}/emoji endpoint to toggle, add, remove message emojis (#925)
* add endpoint to set message emojis

* make refresh result optional in db utils
2023-01-25 09:31:20 +01:00
Andreas KöpfandGitHub ffaf5c48d1 907 avoid duplicate labeling & ranking tasks (#923)
* store message_id in message_reactions and task_id in text_labels

* exclude tasks to which users already responded to

* remove test code

* fix join in find_incomplete_rankings_ex
2023-01-24 17:33:15 +01:00
Andreas KöpfandGitHub 2d21b65ed0 Add lang-tag based task selection (lang-separation) (#863)
* lang based task selection

* use BCP 47 instead of ISO 639-1

* add Field(None, nullable=True)

* update migration script down_revision
2023-01-20 19:58:33 +01:00
Andreas KöpfandGitHub 70fc80aa08 Add keyset pagination for users ordered by username / display_name (#851)
* add keyset pagination for user ordered by username or display_name

* add index on display-name for user table

* update down_revision in migration script
2023-01-20 16:32:13 +01:00
Andreas Köpf 038d1cc004 last_uddated for user_stats, swtich to timestamptz 2023-01-19 22:07:08 +01:00
Andreas KöpfandGitHub e58ffd64fa add /api/v1/users/{user_id}/stats endpoint (#744)
* add /api/v1/users/{user_id}/stats endpoint

* return 0 stats and add /api/v1/users/{user_id}/stats/{time_frame}

* use utcnow() as modified date for 0 stats
2023-01-15 20:24:15 +00:00
Andreas KöpfandGitHub b5bb5bb7c0 Add leaderboard stats, periodic updates via fastapi-utils (#724)
* add leaderboard stats, periodic update via fastapi-utils

* count label tasks for assistant and prompter replies

* Daily stats update every 15 mins, simplify leaderboard endpoint

* add indices for some created_date columns

* make user stats update intervals configurable

* make sure intervals are positive
2023-01-15 12:04:19 +01:00
Andreas Köpf 6547058778 add rank to message table and fix message_toxicity migration 2023-01-14 15:12:10 +01:00
a902c600fa Store Message Toxicity in database (#553)
* [NEW] MessageToxicity table

* [NEW] Alembic message Toxicity

* [NEW] Model name enum

* [NEW] Refactor Enum HF

* [NEW] Settings: DEBUT_SKIP_TOXICITY_CALCULATION

* [NEW] Store toxicity values

* [FIX] Merge conflict

* [FIX] Documentation

* [NEW] save_toxicity: function

* [FIX] Formatted string

* [NEW] DEBUG_SKIP_TOXICITY_CALCULATION=True

* [FIX] HfClassificationModel

* [FIX] Alembic merge heads

* [NEW] Refactor save_toxicity

* [NEW] Separating score/label

* [NEW] Store score and label

* [FIX] Cleaning Alembic

* [NEW] Clean HF names

* [NEW] Not type hinting

* [NEW] Update alembic versions

* [NEW] Revert the changes

* [NEW] Type hinting label & score

* Updated down_revision in migration script

Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-14 12:22:55 +00:00
Andreas Köpf 945c645818 returning list[FrontEndUser] from api/v1/frontend_users/ endpoint 2023-01-12 22:58:43 +01:00
Andreas Köpf f264b43cde fix user's notes column server_default value 2023-01-12 21:43:33 +01:00
050d4902f3 538: Add endpoints to manage users (#601)
* Add endpoints for getting, updating, deleting users by global user ID

* Resolve formatting

* Include alembic revision script

* Updated down_revision to current alembic head

Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-12 21:30:07 +01:00
Andreas Köpf b659e1cf01 remove accepted_messages from message_tree_state 2023-01-12 01:21:32 +01:00
14fa08e2e7 Message tree state machine (#555)
* add query_incomplete_rankings()

* Add SQL queries for TreeManager task selection

* first working version of TreeManager.next_task()

* remove old generate_task(), add mandatory_labels to text_labels task

* Add ConversationMessage list to Ranking tasks

* add more sophisticated sql queries to find extendible trees

* add TreeManager.query_extendible_parents()

* fix task validation, seed data insertion (reviewed)

* provide user for task selection in text-frontend

* enter 'growing' state

* enter 'aborted_low_grade' state

* enter 'ranking' state

* check tree 'growing' state upon relpy insertion

* exclude user from labeling their own messages (added DEBUG_ALLOW_SELF_LABELING setting)

* add DEBUG_ALLOW_SELF_LABELING to docker-compose.yaml

* fix ranking submission

* add query_tree_ranking_results()

* add ranked_message_ids to RankingReactionPayload

* fix reply_messages instead of prompt_messages

* incorment 'ranking_count' of ranked replies

* added logic to check_condition_for_scoring_state

* changes to msg_tree_state_machine

* pre-commit changes

* enter 'ready_for_scoring' state

* re-add HF embedding call (lost during merge)

* use prepare_conversation() helper for seed-data creation

* Partially add user specified task selection

Co-authored-by: Daniel Hug <danielpatrickhug@gmail.com>
2023-01-11 10:54:03 +01:00
Nil-Andreu 70620520b4 [NEW] Created date 2023-01-08 21:29:12 +01:00
jojopirker a677e40cff insert embedding now to new table 2023-01-08 16:46:53 +01:00
jojopirker 11d55d572a message embeddings in Messages table 2023-01-08 12:28:38 +01:00
jojopirker 9194e15b80 bugfix in alembic downgrade (frontend_type not frontend_id) 2023-01-08 10:59:12 +01:00
jojopirkerandGitHub 5fb0121668 #438 Frontend type (#483)
* add dev setup to readme & alembic description

* frontend_type for api_client

* readme update & missing file

* pre-commit
2023-01-07 21:33:57 +01:00
Oliver StanleyandGitHub 69bc799cd9 344: Create tasks for text labels (#381)
* Implement label task for initial prompts and replies

* Resolve formatting

* Include missing argument

* Modify text_labels API to match new model, update DB schema accordingly

* Send valid labels as part of label tasks

* Send correctly formatted valid_labels list

* Fix request format

* Fix request details for text-frontend reply label task

* Include message_id in tasks

* Address review comments

* Fix alembic tree
2023-01-06 18:39:04 +01:00
Daniel HugandGitHub 935e556cf6 Add message tree state model (#414)
* inital commit, created file for MessageTreeState table

* added initial implementation of MessageTreeState SQl model and added it to__init__

* Ran Alembic revision for migration

* reran precommit on

* removed create_data and deleted flag from model

* ran migration revision to remove unwant class variables date_created and deleted

* removed unused imports

* ran pre-commit

* Updated States definiton to be Enums instead fo Constants

* ran pre-commit formatting

* Fixed Enum class

* ran pre-commit
2023-01-05 21:32:47 +01:00
Gareth Davidson 7000e10bc0 apply pre-commit rules 2023-01-02 00:01:45 +00:00
Igor MiagkovandAndreas Köpf 6d98ba1f75 fix alembic migration 2022-12-31 10:31:31 +01:00
Igor MiagkovandAndreas Köpf 13d01b5a2f management api 2022-12-31 10:31:31 +01:00
Andreas KöpfandAndreas Köpf 5f16a8f780 make sure alembic migration works in both directions 2022-12-31 00:18:55 +01:00
Andreas KöpfandAndreas Köpf 048802fa1a add alembic migration script 2022-12-31 00:18:55 +01:00
Igor MiagkovandAndreas Köpf 7867466700 added trusted api client flag 2022-12-30 00:35:30 +01:00
Yannic KilcherandGitHub 3ab2e01b21 Merge pull request #128 from GraemeHarris/add-lang-column
Added column to post table for lang with US english set to default.
2022-12-29 21:33:41 +01:00
Igor Miagkov efafc0173a added support for collective tasks 2022-12-29 21:32:17 +04:00
Graeme Harris e5c32eda21 Updates from linting 2022-12-29 19:26:30 +02:00
GraemeHarrisandGitHub 8046a4a15b Merge branch 'main' into add-lang-column 2022-12-29 09:52:11 +02:00
Graeme Harris 00ebf90871 Added column to post table for lang with US english set to default. Included alembic migration 2022-12-28 23:25:29 +02:00
Igor MiagkovandAndreas Köpf abf2fcc1c8 Migrations for task sampling 2022-12-28 22:23:16 +01:00
Andreas KöpfandAndreas Köpf f74fe68f87 add journal table and JournalWriter helper class 2022-12-27 22:37:48 +01:00
Yannic Kilcher 78a7747e05 added docstring to migration and renamed file 2022-12-25 18:58:48 +01:00
Yannic Kilcher a37bf6bf41 added text labels to the API 2022-12-25 17:08:57 +01:00
Andreas Köpf 8a48722e72 first api-interaction, fix auth_method unique-index 2022-12-22 18:41:50 +01:00
Yannic Kilcher b34ef75423 started re-working python code 2022-12-19 18:00:09 +01:00
Yannic Kilcher 1acdc66973 moved alembic 2022-12-17 23:40:40 +01:00
Andreas Köpf afb5ec3669 change alembic file_template 2022-12-17 23:31:35 +01:00
Andreas Köpf 2a56438a55 add auth_method column to person table 2022-12-17 23:01:21 +01:00
Andreas Köpf 1f31e6a499 add frontend post_id binding and post reply handing 2022-12-16 13:32:06 +01:00
Andreas Köpf b38c14233f add polymorphic JSONB payload support
update nullable fields, new auth check
2022-12-16 02:08:43 +01:00
Andreas Köpf a827b7f19e add sqlmodel classes 2022-12-15 23:51:58 +01:00
Andreas Köpf 7d88ff4a8c use JSONB instead of Text of payload columns 2022-12-15 23:51:58 +01:00
Andreas Köpf e1a7ae42aa add v1 database alembic migration script 2022-12-15 23:51:58 +01:00
Yannic Kilcher 8dd9e16c33 Merge branch 'main' of github.com:LAION-AI/Open-Chat-GPT 2022-12-13 12:39:14 +01:00
Yannic Kilcher f407887501 renamed alemic to alembic 2022-12-13 11:57:54 +01:00