Commit Graph
38 Commits
Author SHA1 Message Date
Andreas KöpfandGitHub da1c81d2c9 Add LabelDescription list to labeling tasks, make +1/-1 emojis exclusive (#947)
* add LabelDescription list to labeling tasks

* make +1 & -1 emoji exclusive (only one of both or none)

* add red_flag emoji to message when reported

* fix task's valid labels

* fix typo
2023-01-27 00:54:29 +01:00
Andreas KöpfandGitHub f3ffde47ff add preferred lonely_children extension (#942)
* add preferred lonely_children extension

* simplify sampling process, lower the probability to 25%

* exclude parents for replies that were recently used

* lonely children := count > 0

* consider only tasks not done for parent exclusion

* increase lonely child sampling probability
2023-01-26 23:00:54 +01:00
Andreas Köpf 348999a936 exclude trees in ranking state in acitve tree count 2023-01-26 19:08:58 +01:00
Andreas Köpf d4688835d5 check condition for scoring on startup 2023-01-26 16:33:03 +01:00
Andreas KöpfandGitHub c2fa476904 Add user emoji augmentation for message queries (#937)
* add disposition to text labeling tasks

* add emoji stats to ConversationMessage

* add user emoji augmentation for message queries

* add auth_method,username to message queries (query emoji status)

* add auth_method+username for single message

* fix param name typo

* only join rows when message.emojis != JSON.NULL

* formatting

* make sure emojis and user_emojis default to {}, []

* remove init_user(), use fresh empty default collections
2023-01-26 14:29:54 +00:00
Andreas Köpf 4d7be8ebb9 wait for pending message reviews before entering ranking state 2023-01-25 18:18:21 +01:00
Andreas Köpf 39a107e851 Use intersection of ranking ID sets, add fetch_siblings() 2023-01-25 16:21:58 +01:00
VechtomovandGitHub 1020dcb024 add retry_scoring_failed_message_trees cli command (#931) 2023-01-25 13:40:36 +01:00
032a748ba5 677 - Add tree message export (#808)
* Added - Basic functions to export trees for users, export-ready trees and specific tree ids to files

* Added print to logger by default for no file specified

* linting to remove extra imports

* Added cli for exporting trees which are ready to export

Fixed some accidental removal

Updated message lookup to use dict for better perf

* removed unused imports

* changed export flag for including deleted prompts back to include_deleted for better understandability

* Use native collection types list, tuple, dict

* pre-commit fix

Co-authored-by: Andreas Köpf <andreas.koepf@provisio.com>
2023-01-24 22:13:10 +01:00
Andreas KöpfandGitHub ffaf5c48d1 907 avoid duplicate labeling & ranking tasks (#923)
* store message_id in message_reactions and task_id in text_labels

* exclude tasks to which users already responded to

* remove test code

* fix join in find_incomplete_rankings_ex
2023-01-24 17:33:15 +01:00
Andreas Köpf d72f7771ca fix HF detoxify call 2023-01-24 17:26:19 +01:00
Ori YonayandGitHub cec49614c2 added message size limit of 2000 characters (issue 822) (#880)
* added message size limit (issue 822)
2023-01-21 22:37:42 +01:00
Andreas KöpfandGitHub 2d21b65ed0 Add lang-tag based task selection (lang-separation) (#863)
* lang based task selection

* use BCP 47 instead of ISO 639-1

* add Field(None, nullable=True)

* update migration script down_revision
2023-01-20 19:58:33 +01:00
Andreas Köpf 7a1a8c8b54 fetch whole message tree for purge (including non-reviewed & deleted) 2023-01-19 16:42:13 +01:00
Andreas Köpf 1a93c21f94 add logging 2023-01-19 16:28:03 +01:00
Andreas Köpf 547e355e27 add update_leaderboard..(), admin/purge/{user-id}/messages, ban param for purge_user() 2023-01-19 15:10:22 +01:00
Andreas KöpfandGitHub 335af5d641 add admin purge user function (#834)
* add admin purge user function

* improve comments

* minor naming changes

* ensuer user is enabled for tasks api requests

* add preview with stats to /admin/purge_user/{id} endpoint

* add update_children_counts()
2023-01-19 12:33:55 +01:00
Andreas Köpf 7e8d4f5a6e fix only_active condition of tree_message_count_stats() 2023-01-18 22:02:35 +01:00
Andreas KöpfandGitHub 718faa098e Add combined TreeManager stats endpoint (#816) 2023-01-18 00:54:19 +01:00
Andreas Köpf acaa56ebb0 fix ranking assert for 1st level children 2023-01-18 00:49:28 +01:00
Andreas Koepf 0f896d910e make sure we enter READY_FOR_EXPORT after ranking 2023-01-17 17:50:17 +00:00
Andreas Köpf 9b6dc6bae2 fix join types in TreeManager 2023-01-17 11:01:36 +01:00
Andreas Köpf 2ee01d1224 fix query in TreeManager.query_extendible_parents() 2023-01-17 10:45:52 +01:00
Andreas Köpf d85cec3d2f extensible -> extendible 2023-01-17 10:21:57 +01:00
Andreas Köpf 6e4d8097b6 fix availability for label_prompter_reply 2023-01-17 10:15:30 +01:00
Andreas Köpf 988913943f use '.value' of str-enums in db queries 2023-01-17 09:54:20 +01:00
8b30c7b68e add changes for auto_main, tree_manager, and utils/ranking (#786)
* add changes for auto_main, tree_manager, and utils/ranking

* pre-commit changes

Co-authored-by: Alexander Mattick <alex.mattick@fau.de>
2023-01-17 08:27:21 +01:00
Andreas Köpf 2d4e39cf5d use sqlalchemy for query_reviews_for_message() in TreeManager 2023-01-16 20:31:29 +01:00
Andreas KöpfandGitHub ead51ff423 770 tree manager allow to specify desired task_type no prompter ranking (#775)
* only ranking assistant replies by default

* add tasks/availability endpoint allow to specify desired task

* move rank_prompter_replies option to TreeManagerConfiguration

* fix type annotation

* remove desired_task_type from _random_task_selection()

* fix typo

* Convert query_tree_size to sqlachemy, return 'full' text-labeling tasks if they were explicitly requested
2023-01-16 20:05:40 +01:00
c6fbf5543b 599 add row versioning to backend tables (#710)
* fix: isolation level and nested db.commit() with retry wrappers on concurrent update errors

* refactor: incorporated review comments

changes decorator methods to managed_tx_method and async_managed_tx_method
new enum CommitMode
removed commented commit() from the previous commits

* fix: merge pre-commit errors

* fix: merge pre-commit changes

* fix: conflict in existing OasstErrorCode

* refactor: Added a refresh just to be sure that the select command is triggered on the mapped object

* fix: added refresh for async decorator

Co-authored-by: James Melvin <melvin@gameface.ai>
2023-01-16 08:43:07 +01:00
Andreas KöpfandGitHub b5bb5bb7c0 Add leaderboard stats, periodic updates via fastapi-utils (#724)
* add leaderboard stats, periodic update via fastapi-utils

* count label tasks for assistant and prompter replies

* Daily stats update every 15 mins, simplify leaderboard endpoint

* add indices for some created_date columns

* make user stats update intervals configurable

* make sure intervals are positive
2023-01-15 12:04:19 +01:00
a902c600fa Store Message Toxicity in database (#553)
* [NEW] MessageToxicity table

* [NEW] Alembic message Toxicity

* [NEW] Model name enum

* [NEW] Refactor Enum HF

* [NEW] Settings: DEBUT_SKIP_TOXICITY_CALCULATION

* [NEW] Store toxicity values

* [FIX] Merge conflict

* [FIX] Documentation

* [NEW] save_toxicity: function

* [FIX] Formatted string

* [NEW] DEBUG_SKIP_TOXICITY_CALCULATION=True

* [FIX] HfClassificationModel

* [FIX] Alembic merge heads

* [NEW] Refactor save_toxicity

* [NEW] Separating score/label

* [NEW] Store score and label

* [FIX] Cleaning Alembic

* [NEW] Clean HF names

* [NEW] Not type hinting

* [NEW] Update alembic versions

* [NEW] Revert the changes

* [NEW] Type hinting label & score

* Updated down_revision in migration script

Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-14 12:22:55 +00:00
Andreas Koepf d0ef7c4bbc add missing doc-strings for tree-manager mandatory labels settings 2023-01-13 16:04:06 +00:00
Andreas Koepf 343853fb57 add TreeManagerConfiguration as sub-module to global pydantic settings class 2023-01-13 15:52:09 +00:00
Andreas Köpf 7b78430569 remove accepted_messages ref 2023-01-12 01:23:05 +01:00
Andreas Köpf 025d20e638 shuffle ranking options, add simple mode for text-labels 2023-01-11 23:47:56 +01:00
Andreas Köpf 0117159da0 returning 503 Service Unavailable when requested task type is not available 2023-01-11 13:56:21 +01:00
14fa08e2e7 Message tree state machine (#555)
* add query_incomplete_rankings()

* Add SQL queries for TreeManager task selection

* first working version of TreeManager.next_task()

* remove old generate_task(), add mandatory_labels to text_labels task

* Add ConversationMessage list to Ranking tasks

* add more sophisticated sql queries to find extendible trees

* add TreeManager.query_extendible_parents()

* fix task validation, seed data insertion (reviewed)

* provide user for task selection in text-frontend

* enter 'growing' state

* enter 'aborted_low_grade' state

* enter 'ranking' state

* check tree 'growing' state upon relpy insertion

* exclude user from labeling their own messages (added DEBUG_ALLOW_SELF_LABELING setting)

* add DEBUG_ALLOW_SELF_LABELING to docker-compose.yaml

* fix ranking submission

* add query_tree_ranking_results()

* add ranked_message_ids to RankingReactionPayload

* fix reply_messages instead of prompt_messages

* incorment 'ranking_count' of ranked replies

* added logic to check_condition_for_scoring_state

* changes to msg_tree_state_machine

* pre-commit changes

* enter 'ready_for_scoring' state

* re-add HF embedding call (lost during merge)

* use prepare_conversation() helper for seed-data creation

* Partially add user specified task selection

Co-authored-by: Daniel Hug <danielpatrickhug@gmail.com>
2023-01-11 10:54:03 +01:00