wassname/Open-Assistant

mirror of https://github.com/wassname/Open-Assistant.git synced 2026-06-27 16:10:30 +08:00

Files

T

History

Yannic Kilcher b6a0eedf81 various fixes to the SFT code

2023-02-11 21:37:35 +01:00

..

__init__.py

os private dataset

2023-02-11 13:20:42 +01:00

dialogue_collator.py

pre commits

2023-02-11 10:29:56 +01:00

formatting.py

os private dataset

2023-02-11 13:20:42 +01:00

prompt_dialogue.py

various fixes to the SFT code

2023-02-11 21:37:35 +01:00

qa_datasets.py

[fix] Fix other PR merge bug

2023-02-11 01:53:37 +00:00

README.md

Fix typos (#1143 )

2023-02-05 20:18:03 +01:00

summarization.py

[feature] Add mix conversation augmentation

2023-02-01 22:14:11 +00:00

toxic_conversation.py

[fix] prosocial dialogue format error

2023-01-22 14:00:20 +00:00

translation.py

Get working on multi-gpu

2023-02-08 20:49:25 +00:00

README.md

Dataset collections overview:

currently dataset can be divided into 3 classes

language knowledge
- summarization
- translation
dialogue : don't let user know you are a robot
STEM : knowledge about the world
- coding
- world knowledge <= ideally we want to handle this via prefix context

Issues and TODO:

as dataset are growing, how can we update this section less
ideally we can update the config yaml and new dataset will be download from hub
- one possible idea is we upload the transform format of these dataset to the OA hub