Open-Assistant/notebooks/data-argumentation/UnifiedQA.ipynb

UnifiedQA Dataset

The purpose of this notebook is to download datasets from the UnifiedQA dataset collection and convert them into a format that can be used for training the OpenAssistant.

The UnifiedQA repo can be found here: https://github.com/allenai/unifiedqa

If you extend or use this work, please cite the relevant papers:

@article{khashabi2022unifiedqa,
    title={UnifiedQA-v2: Stronger Generalization via Broader Cross-Format Training},
    author={Khashabi, Daniel and Kordi, Yeganeh and Hajishirzi, Hannaneh},
    journal={arXiv preprint arXiv:2202.12359},
    year={2022}
}

Compare xP3 and UnifiedQA

As many of the datasets that are in UnifiedQA are already in xP3, we do a simple (and incomplete) check to limit the number of datasets that we download.

In [82]:

xp3_list = [
    "Code Miscellaneous",
    "CodeComplex",
    "Docstring Corpus",
    "GreatCode",
    "State Changes",
    "Closed-book QA",
    "Hotpot QA",
    "Trivia QA",
    "Web Questions",
    "Wiki QA",
    "Extractive QA",
    "Adversarial QA",
    "CMRC2018",
    "DRCD",
    "DuoRC",
    "MLQA",
    "Quoref",
    "ReCoRD",
    "ROPES",
    "SQuAD v2",
    "xQuAD",
    "TyDI QA",
    "Primary",
    "Goldp",
    "Multiple-Choice QA",
    "ARC",
    "C3",
    "CoS-E",
    "Cosmos",
    "DREAM",
    "MultiRC",
    "OpenBookQA",
    "PiQA",
    "QUAIL",
    "QuaRel",
    "QuaRTz",
    "QASC",
    "RACE",
    "SciQ",
    "Social IQA",
    "Wiki Hop",
    "WiQA",
    "Paraphrase Identification",
    "MRPC",
    "PAWS",
    "PAWS-X",
    "QQP",
    "Program Synthesis",
    "APPS",
    "CodeContests",
    "JupyterCodePairs",
    "MBPP",
    "NeuralCodeSearch",
    "XLCoST",
    "Structure-to-text",
    "Common Gen",
    "Wiki Bio",
    "Sentiment",
    "Amazon",
    "App Reviews",
    "IMDB",
    "Rotten Tomatoes",
    "Yelp",
    "Simplification",
    "BiSECT",
    "Summarization",
    "CNN Daily Mail",
    "Gigaword",
    "MultiNews",
    "SamSum",
    "Wiki-Lingua",
    "XLSum",
    "XSum",
    "Topic Classification",
    "AG News",
    "DBPedia",
    "TNEWS",
    "TREC",
    "CSL",
    "Translation",
    "Flores-200",
    "Tatoeba",
    "Word Sense disambiguation",
    "WiC",
    "XL-WiC",
    "Evaluation datasets (included in xP3all except for HumanEval)",
    "Natural Language Inference",
    "ANLI",
    "CB",
    "RTE",
    "XNLI",
    "Coreference Resolution",
    "Winogrande",
    "XWinograd",
    "Program Synthesis",
    "HumanEval",
    "Sentence Completion",
    "COPA",
    "Story Cloze",
    "XCOPA",
    "XStoryCloze",
    "Additional xP3all datasets",
    "Coreference Resolution",
    "WSC (Fixed)",
    "Sentence Completion",
    "HellaSwag",
    "Translation",
    "MultiEurlex",
]

In [83]:

unifiedQA_list = [
    "SQuAD 1.1",
    "SQuAD 2",
    "NewsQA",
    "Quoref",
    "ROPES",
    "NarrativeQA",
    "DROP",
    "NaturalQuestions",
    "MCTest",
    "RACE",
    "OpenBookQA",
    "ARC",
    "CommonsenseQA",
    "QASC",
    "PhysicalIQA",
    "SocialIQA",
    "Winogrande",
    "BoolQ",
    "MultiRC (yes/no)",
    "BoolQ-NP",
]

Now that we've defined the list of datasets (which we found in the paper for UnifiedQA and on the Hugging Face page of xP3) we can do the simple check.

In [84]:

for ds in unifiedQA_list:
    if ds not in xp3_list:
        print(ds)

SQuAD 1.1
SQuAD 2
NewsQA
NarrativeQA
DROP
NaturalQuestions
MCTest
CommonsenseQA
PhysicalIQA
SocialIQA
BoolQ
MultiRC (yes/no)
BoolQ-NP

The SQuAD dataset is actually covered (with a slightly different name) but the other ones should be downloaded.

OpenAssistant Data Scheme

We will use the data scheme that can be found in the docs for Open-Assistant. This code is taken from the StackExchange notebook.

In [85]:

from typing import TypeVar, List, Dict, Any, Literal
from json import JSONEncoder

T = TypeVar("T", bound="ConversationTreeNode")


class ConversationTreeNode:
    text: str  # The text of the node
    role: Literal["prompter", "assistant"]  # Whether the node is a user prompt/follow-up or an assistant response
    children: List[T]  # The children of the node (if you have a linear conversation, this will be of length 0 or 1)
    metadata: Dict[str, Any]  # Node metadata (see below)

    def __init__(
        self, text: str, role: Literal["prompter", "assistant"], children: List[T], metadata: Dict[str, Any]
    ) -> None:
        self.text = text
        self.role = role
        self.children = children
        self.metadata = metadata


class ConversationTree:
    root: ConversationTreeNode  # The node containing the initial prompt
    metadata: Dict[str, Any]  # Tree metadata, different from root node metadata.

    def __init__(self, root: ConversationTreeNode, metadata: Dict[str, Any]) -> None:
        self.root = root
        self.metadata = metadata


# subclass JSONEncoder
class TreeEncoder(JSONEncoder):
    def default(self, o):
        return o.__dict__

Download and convert

We firstly import pandas, which we'll use to download the TSV files from Google Cloud Storage, and any other libraries that we'll need.

In [86]:

import pandas as pd
import json

The following is a simple function to take the data (which has two columns) and convert it to a tree with a root note (question) and one child (answer).

In [87]:

def convert_unified_qa(dataset_url):
    # download using pandas
    ds = pd.read_csv(dataset_url, on_bad_lines="skip", names=["Question", "Answer"], sep="\t")
    # get name for metatdata
    ds_name = dataset_url.split("/unifiedqa/data/")[1].split("/")[0]

    # create conversation forest
    conversation_forest = []
    for item in ds.itertuples():
        # build nodes and tree
        root = ConversationTreeNode(text=item.Question, role="prompter", children=[], metadata=None)
        child = ConversationTreeNode(text=item.Answer, role="assistant", children=[], metadata=None)
        root.children.append(child)
        conversation_tree = ConversationTree(root=root, metadata={"dataset": ds_name})

        conversation_forest.append(conversation_tree)

    conversation_forest_json = [
        json.loads(TreeEncoder().encode(conversation_tree)) for conversation_tree in conversation_forest
    ]

    print(json.dumps(conversation_forest_json, indent=4), file=open(f"./{ds_name}.json", "w+"))

    print("*****", ds_name, "****")
    print(ds.head(2))
    print("....")

We now define the list of URLs that we want to download. These URLs were found by manually going UnifiedQA'S Google Cloud bucket: https://console.cloud.google.com/storage/browser/unifiedqa/data

In [88]:

urls = [
    "https://storage.googleapis.com/unifiedqa/data/natural_questions/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/narrativeqa/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/newsqa/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/drop/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/commonsenseqa/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/physical_iqa/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/social_iqa/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/boolq/train.tsv",
    "https://storage.googleapis.com/unifiedqa/data/boolq_np/train.tsv",
]

In [77]:

for url in urls:
    convert_unified_qa(url)

***** natural_questions ****
                                            Question  \
0  which is the most common use of opt-in e-mail ...   
1           how i.met your mother who is the mother?   

                                              Answer  
0  a newsletter sent to an advertising firm's cus...  
1                                    Tracy McConnell  
....
***** narrativeqa ****
                                            Question  \
0  Who is Miss Delmer? \n  At Madeline Hall, an o...   
1  Who is Miss Delmer? \n  At Madeline Hall, an o...   

                                              Answer  
0   the elderly spinster aunt of the Earl de Vers...  
1                      She's Captail Delmar's aunt.   
....
***** newsqa ****
                                            Question      Answer
0  How many Americans are part of the federal foo...  31 million
1  How much did Sean Callebs live on? \n (CNN) --...        $176
....
***** drop ****
                                            Question Answer
0  How many points did the buccaneers need to tie...      3
1  How many field goals did the Lions score? \n T...      2
....
***** commonsenseqa ****
                                            Question           Answer
0  The sanctions against the school were a punish...           ignore
1  Sammy wanted to go to where the people were.  ...  populated areas
....
***** physical_iqa ****
                                            Question  \
0  When boiling butter, when it's ready, you can ...   
1  To permanently attach metal legs to a chair, y...   

                                              Answer  
0                                 Pour it into a jar  
1   Weld the metal together to get it to stay fir...  
....
***** social_iqa ****
                                            Question           Answer
0  How would Others feel as a result? \n (A) like...   like attending
1  What will Others want to do next? \n (A) disag...      get to work
....
***** boolq ****
                                            Question Answer
0  can you buy beer at a grocery store in pa? \n ...    yes
1  is the baby in fuller house a twin? \n (List o...    yes
....
***** boolq_np ****
                                            Question Answer
0  do iran and afghanistan write the same languag...    yes
1  do iran and afghanistan read the same language...    yes
....

In [ ]: