Commit Graph

305 Commits

Author SHA1 Message Date
Andreas Köpf eb4c41e3c6 send full conversation (including last-message) in label tasks 2023-01-28 20:23:39 +01:00
Andreas Köpf 54503b7e1b reduce userstats cron defaults, fix reference error 2023-01-28 19:10:03 +01:00
Andreas Köpf 19116f7251 add optional message_id query param to text_labels/valid_labels endpoint 2023-01-28 15:29:38 +01:00
Andreas Köpf 264e914225 exclude fails_task from default valid labels 2023-01-28 15:07:46 +01:00
Andreas Köpf c8d16285d0 Import message trees from jsonl file (#964)
* add new backlog_ranking tree state

* add first version of import script

* allow activation of trees during import

* add min_active_rankings_per_lang config param

* add settings docstring
2023-01-28 15:05:46 +01:00
MattAlexMiracle 994dce7c76 language classifier (#963)
Co-authored-by: Alexander Mattick <alex.mattick@fau.de>
2023-01-28 00:56:56 +01:00
Andreas Köpf 0903ec5d58 fix quility label addition 2023-01-28 00:03:52 +01:00
Andreas Köpf 8ea53a54eb add quality to valid-labels for simple text labeling tasks (anti-spam) 2023-01-28 00:01:49 +01:00
dhug 49b5999ce6 Duplicate message reply filter (#958)
* added changes for user specific message dumplication filter, added error codes, and settings variable as described in Draft PR #926, ran precommit

* removed debug statements

* add missing await to async_managed_tx_method

* add 2nd missing await to async_managed_tx_method

* added changes for user specific message dumplication filter, added error codes, and settings variable as described in Draft PR #926, ran precommit

* removed debug statements

* assert task user matches prompt_repository user

* removed assert statments

* moved duplicate_message_filter and message_length check into store_text_reply

* removed old checks in tree_manager
2023-01-27 21:52:19 +00:00
Andreas Köpf 3a32a10b23 add reply_message to LabelConversationReplyTask & LabelConversationReplyPayload 2023-01-27 22:30:27 +01:00
Andreas Köpf 4b3ce24c72 add parsing of additional x-oasst-user header field (#957)
* add parsing of additional x-oasst-user header field

* simplify passing FrontendUserId to PromptRepository, use frontend_user_id on more message enpoints

* remove unrelated change
2023-01-27 21:28:57 +01:00
Andreas Köpf 002f847406 add missing alembic migration script 2023-01-27 21:16:31 +01:00
Andreas Köpf 9bad3d6e74 Exclude certain users from leaderboard stats (#961)
Co-authored-by: Akhil Datla <66145155+akhil-datla@users.noreply.github.com>
2023-01-27 21:04:55 +01:00
Andreas Köpf c7692b9049 add 2nd missing await to async_managed_tx_method 2023-01-27 19:44:48 +01:00
Andreas Köpf d165987256 add missing await to async_managed_tx_method 2023-01-27 19:43:08 +01:00
James Melvin Ebenezer 3b04080d7b 949_transaction error handling (#950)
* fix: transaction error handling

* refactor: retry handling for all decorators as per review comments

* fix: raising retry exhausted error

* fix: avoid auto refresh on RollBack and review comments

* removed refresh_result param from managed_tx_function

---------

Co-authored-by: James Melvin <melvin@gameface.ai>
Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-27 18:06:25 +01:00
Andreas Köpf da1c81d2c9 Add LabelDescription list to labeling tasks, make +1/-1 emojis exclusive (#947)
* add LabelDescription list to labeling tasks

* make +1 & -1 emoji exclusive (only one of both or none)

* add red_flag emoji to message when reported

* fix task's valid labels

* fix typo
2023-01-27 00:54:29 +01:00
Andreas Köpf f3ffde47ff add preferred lonely_children extension (#942)
* add preferred lonely_children extension

* simplify sampling process, lower the probability to 25%

* exclude parents for replies that were recently used

* lonely children := count > 0

* consider only tasks not done for parent exclusion

* increase lonely child sampling probability
2023-01-26 23:00:54 +01:00
Andreas Köpf 348999a936 exclude trees in ranking state in acitve tree count 2023-01-26 19:08:58 +01:00
Andreas Köpf d4688835d5 check condition for scoring on startup 2023-01-26 16:33:03 +01:00
Andreas Köpf c2fa476904 Add user emoji augmentation for message queries (#937)
* add disposition to text labeling tasks

* add emoji stats to ConversationMessage

* add user emoji augmentation for message queries

* add auth_method,username to message queries (query emoji status)

* add auth_method+username for single message

* fix param name typo

* only join rows when message.emojis != JSON.NULL

* formatting

* make sure emojis and user_emojis default to {}, []

* remove init_user(), use fresh empty default collections
2023-01-26 14:29:54 +00:00
MattAlexMiracle 5d4f74f9d6 Ranked pairs (#933)
* commented out legacy numerical solver

* added comments and task_scheduling for selecting which task to serve to users

* removed standalone task weighting

* pre-commit hook rerun

* fixed ranking

* fix index error

* ranking fix

* fix typo

Co-authored-by: Alexander Mattick <alex.mattick@fau.de>
Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-26 09:50:25 +00:00
Andreas Köpf 4d7be8ebb9 wait for pending message reviews before entering ranking state 2023-01-25 18:18:21 +01:00
Andreas Köpf 39a107e851 Use intersection of ranking ID sets, add fetch_siblings() 2023-01-25 16:21:58 +01:00
Vechtomov 1020dcb024 add retry_scoring_failed_message_trees cli command (#931) 2023-01-25 13:40:36 +01:00
Andreas Köpf 558b207013 Add /messages/{message_id}/emoji endpoint to toggle, add, remove message emojis (#925)
* add endpoint to set message emojis

* make refresh result optional in db utils
2023-01-25 09:31:20 +01:00
Graeme Harris 032a748ba5 677 - Add tree message export (#808)
* Added - Basic functions to export trees for users, export-ready trees and specific tree ids to files

* Added print to logger by default for no file specified

* linting to remove extra imports

* Added cli for exporting trees which are ready to export

Fixed some accidental removal

Updated message lookup to use dict for better perf

* removed unused imports

* changed export flag for including deleted prompts back to include_deleted for better understandability

* Use native collection types list, tuple, dict

* pre-commit fix

Co-authored-by: Andreas Köpf <andreas.koepf@provisio.com>
2023-01-24 22:13:10 +01:00
Andreas Köpf ffaf5c48d1 907 avoid duplicate labeling & ranking tasks (#923)
* store message_id in message_reactions and task_id in text_labels

* exclude tasks to which users already responded to

* remove test code

* fix join in find_incomplete_rankings_ex
2023-01-24 17:33:15 +01:00
Andreas Köpf d72f7771ca fix HF detoxify call 2023-01-24 17:26:19 +01:00
Andreas Köpf 50421dfada retry user lookup in case of UniqueViolation (ix_user_username conflict) 2023-01-24 09:57:45 +01:00
Jordi Smit 8c632f9ef9 add lang filter option to message endpoints (#902)
* add lang filter option to message endpoints

* add lang filter option to frontend_users endpoints

* move lang param before api_client

Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-23 21:46:59 +00:00
Andreas Köpf b7fab83d36 remove unused crud baseclass 2023-01-23 20:49:50 +01:00
Andreas Köpf 43e7d4846b lt->before, gt->after for cursor endpoints 2023-01-22 18:43:38 +01:00
Andreas Köpf 0b8ed52102 fix desc ordering of message cursor endpoints 2023-01-22 18:37:09 +01:00
notmd 7025b174d2 Merge branch 'main' into 766_admin_enhancement 2023-01-22 17:22:31 +07:00
James Melvin Ebenezer c0391a6df9 fix: redundant row updates with no Task id in text_labels table (#876)
* fix: redundant row updates with no Task id in text_labels table

* fix: review comments incorporated

* fix: better error handling and function name

* fix: review comments

Co-authored-by: James Melvin <melvin@gameface.ai>
2023-01-22 11:08:02 +01:00
Andreas Köpf 28089d9ecf fix username+auth combo check 2023-01-22 09:29:21 +01:00
notmd d9205bec52 Merge branch 'main' into 766_admin_enhancement 2023-01-22 14:58:15 +07:00
Andreas Köpf 3b5b6669a5 move lt-desc order to users-cursor function 2023-01-22 08:42:38 +01:00
notmd 101f2c536a revert change in user_repository 2023-01-22 14:20:39 +07:00
notmd 6945cc5fe7 remove reverse method 2023-01-22 14:14:41 +07:00
notmd fa5702ac9e Merge branch 'main' into 766_admin_enhancement 2023-01-22 14:07:19 +07:00
Andreas Köpf 76b5bdc5d7 fix cursor queries query_users_ordered_by_username/query_users_ordered_by_display_name 2023-01-22 00:58:03 +01:00
Ori Yonay cec49614c2 added message size limit of 2000 characters (issue 822) (#880)
* added message size limit (issue 822)
2023-01-21 22:37:42 +01:00
notmd 0cc6b3b8ae Merge branch 'main' into 766_admin_enhancement 2023-01-21 22:10:15 +07:00
Andreas Köpf e08b36e675 imporve prev/next status for user cursor 2023-01-21 16:04:43 +01:00
notmd 27e1e549c4 fix query in backward direction 2023-01-21 20:10:19 +07:00
Andreas Köpf 186aabe3a5 improve prev,next cursor values 2023-01-21 12:19:45 +01:00
Andreas Köpf 94e5d50537 add users/cursor keyset_pagination endpoint (#866)
* add users/cursor endpoint

* add messages/cursor endpoint

* add user/{user_id}/messages/cursor, frontend_user/{auth_method}/{username}/messages/cursor

* user regex to parse cursor value
2023-01-21 00:29:53 +01:00
Andreas Köpf 2d21b65ed0 Add lang-tag based task selection (lang-separation) (#863)
* lang based task selection

* use BCP 47 instead of ISO 639-1

* add Field(None, nullable=True)

* update migration script down_revision
2023-01-20 19:58:33 +01:00