63 Commits

Author SHA1 Message Date
Andreas Köpf cbc3eba37d add env for mods 2023-02-10 09:11:15 +01:00
Andreas Köpf dfeb969884 move shm_size setting to playbook 2023-02-10 08:27:32 +01:00
Kian-Meng Ang 1e321a6fca Fix typos (#1143)
Found via `codespell -S .mypy_cache,yarn.lock,*.json,*.ipynb -L
rouge,nam,vie`
2023-02-05 20:18:03 +01:00
Yannic Kilcher c878d12069 added backend url 2023-02-05 01:50:30 +01:00
Yannic Kilcher 26f3fd4453 added maintenance mode variable 2023-02-05 01:34:34 +01:00
Andreas Köpf 100ecbe2d5 Limit initial prompts, ensure max_active_trees = growing trees 2023-02-04 23:23:16 +01:00
Andreas Köpf f02543b36f reduce INFO logging 2023-02-04 20:26:20 +01:00
Andreas Köpf 6681aedd01 add P_LONELY_CHILD_EXTENSION to workflow vars 2023-02-03 09:36:44 +00:00
Andreas Köpf 03321ccc72 add lonely_children_count, p_activate_backlog_tree & min_active_rankings_per_lang to gh workflow config 2023-02-03 09:17:42 +00:00
Andreas Köpf d5c0adba5f fix typo SERCERT -> SECRET 2023-02-02 22:10:50 +00:00
Andreas Köpf 776584503f add new cfg env vars 2023-02-02 22:10:14 +01:00
Andreas Köpf d67b0ad0be add stats intervals & max children count to ansible/workflows 2023-01-29 11:06:32 +01:00
Yannic Kilcher c05ab42e94 added skip toxicity to playbook 2023-01-18 08:53:01 +01:00
Yannic Kilcher 18e7eb8876 updated deploy playbook with env variables 2023-01-17 22:23:33 +01:00
Yannic Kilcher 1b41f8bbfc fixed pre-commit 2023-01-17 12:22:10 +01:00
Yannic Kilcher db8fedf010 added region to aws 2023-01-17 12:02:10 +01:00
Yannic Kilcher 137f733fef added oasst postgres 2023-01-17 11:30:52 +01:00
Yannic Kilcher cd63df1cb2 updated prod playbook 2023-01-16 23:36:33 +01:00
Yannic Kilcher 4eb97daee0 updated prod playbook 2023-01-16 23:32:09 +01:00
Yannic Kilcher b0ce816e15 updated prod playbook 2023-01-16 23:28:30 +01:00
Yannic Kilcher 1facc25b9b updated prod playbook 2023-01-16 23:25:16 +01:00
Yannic Kilcher c60886f55a updated prod playbook 2023-01-16 23:13:25 +01:00
Yannic Kilcher 38da3a6fd5 added pgbackrest config 2023-01-16 23:08:42 +01:00
Yannic Kilcher 6ccbd38462 fixed deploy playbook 2023-01-16 20:19:37 +01:00
Yannic Kilcher 0efade980d fixed staging deployment 2023-01-16 00:22:59 +01:00
Yannic Kilcher 5d441b1570 prepared playbook for prod 2023-01-15 23:30:16 +01:00
Yannic Kilcher cc03376d86 added root tokens and endpoint for adding api keys (#742)
* added root tokens and endpoint for adding api keys

* Change down revision to current alembic head

* removed added_by_root_token

* refactored description

* fixed jinja errors

Co-authored-by: Andreas Köpf <andreas.koepf@provisio.com>
2023-01-15 21:24:25 +00:00
Yannic Kilcher 96307ed4c1 adding modes to ansible 2023-01-15 15:25:21 +01:00
Yannic Kilcher bf66b2a890 fixed workflow 2023-01-14 22:10:39 +01:00
Yannic Kilcher f20594f52f fixed workflow 2023-01-14 22:02:13 +01:00
Yannic Kilcher 6430112dfc fixed workflow 2023-01-14 21:47:35 +01:00
Yannic Kilcher 17920e4e66 fixed workflow 2023-01-14 21:39:10 +01:00
Yannic Kilcher ddae13cf3b fixed workflow 2023-01-14 20:24:23 +01:00
Yannic Kilcher 39cf2cf6db added deploy workflow 2023-01-14 20:06:27 +01:00
Nil Andreu a902c600fa Store Message Toxicity in database (#553)
* [NEW] MessageToxicity table

* [NEW] Alembic message Toxicity

* [NEW] Model name enum

* [NEW] Refactor Enum HF

* [NEW] Settings: DEBUT_SKIP_TOXICITY_CALCULATION

* [NEW] Store toxicity values

* [FIX] Merge conflict

* [FIX] Documentation

* [NEW] save_toxicity: function

* [FIX] Formatted string

* [NEW] DEBUG_SKIP_TOXICITY_CALCULATION=True

* [FIX] HfClassificationModel

* [FIX] Alembic merge heads

* [NEW] Refactor save_toxicity

* [NEW] Separating score/label

* [NEW] Store score and label

* [FIX] Cleaning Alembic

* [NEW] Clean HF names

* [NEW] Not type hinting

* [NEW] Update alembic versions

* [NEW] Revert the changes

* [NEW] Type hinting label & score

* Updated down_revision in migration script

Co-authored-by: Andreas Köpf <andreas.koepf@xamla.com>
2023-01-14 12:22:55 +00:00
Andreas Köpf 14fa08e2e7 Message tree state machine (#555)
* add query_incomplete_rankings()

* Add SQL queries for TreeManager task selection

* first working version of TreeManager.next_task()

* remove old generate_task(), add mandatory_labels to text_labels task

* Add ConversationMessage list to Ranking tasks

* add more sophisticated sql queries to find extendible trees

* add TreeManager.query_extendible_parents()

* fix task validation, seed data insertion (reviewed)

* provide user for task selection in text-frontend

* enter 'growing' state

* enter 'aborted_low_grade' state

* enter 'ranking' state

* check tree 'growing' state upon relpy insertion

* exclude user from labeling their own messages (added DEBUG_ALLOW_SELF_LABELING setting)

* add DEBUG_ALLOW_SELF_LABELING to docker-compose.yaml

* fix ranking submission

* add query_tree_ranking_results()

* add ranked_message_ids to RankingReactionPayload

* fix reply_messages instead of prompt_messages

* incorment 'ranking_count' of ranked replies

* added logic to check_condition_for_scoring_state

* changes to msg_tree_state_machine

* pre-commit changes

* enter 'ready_for_scoring' state

* re-add HF embedding call (lost during merge)

* use prepare_conversation() helper for seed-data creation

* Partially add user specified task selection

Co-authored-by: Daniel Hug <danielpatrickhug@gmail.com>
2023-01-11 10:54:03 +01:00
Yannic Kilcher ba12c35e22 Merge pull request #540 from jojopirker/messageEmbeddings
Store Message embedding
2023-01-09 21:22:56 +01:00
Keith Stevens 72eb6c006d Merge pull request #562 from LAION-AI/372-web-on-dev
Update the dev deployment to use secrets #372
2023-01-09 21:21:21 +09:00
Nil-Andreu ef7bd89df2 [NEW] ansible: DEBUG_SKIP_EMBEDDING_COMPUTATION 2023-01-09 09:09:43 +01:00
Keith Stevens c1b3029f8b Deleting the maildev stanza now that we're using a real email server 2023-01-09 15:46:45 +09:00
Keith Stevens 0e9fd9a140 Seeing the admin users, email from, and next auth secret secrets 2023-01-09 14:56:13 +09:00
Keith Stevens 00e51af787 Adding the email secrets and environment variables 2023-01-09 14:43:45 +09:00
Keith Stevens f22abd524b Adding discord credentials to dev deployment 2023-01-09 14:16:49 +09:00
Keith Stevens 310763ebb8 Adding discord credentials to dev deployment 2023-01-09 14:07:30 +09:00
Keith Stevens d2dd96b43a Wrapping numbers in quotes after learning ansible 2023-01-09 13:47:01 +09:00
Keith Stevens 491d8e7413 Changing the email provider values 2023-01-09 13:39:49 +09:00
d_auras c1dab2d213 deleted my remote.inventory.ini 2023-01-06 21:46:51 +01:00
d_auras 2870524aa7 ansible copies redis.conf to managed node now 2023-01-06 20:03:15 +01:00
d_auras a40e6ec31d ran pre-commit again 2023-01-05 16:25:58 +01:00
rasdani d379193bed add REDIS_HOST environment variable to backend
to comply fully with `docker-compose.yaml`
2023-01-05 16:15:20 +01:00