{ "cells": [ { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "# Testing" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "```source: this key separates the various keys found in the table in Sources. Here's the set of sources with their corresponding value name:```\n", "```\n", "'https://aipulse.org'\n", "'ebook'\n", "'https://qualiacomputing.com'\n", "'alignment forum'\n", "'lesswrong'\n", "'manual'\n", "'arxiv'\n", "'https://deepmindsafetyresearch.medium.com/'\n", "'waitbutwhy.com'\n", "'GitHub'\n", "'https://aiimpacts.org'\n", "'arbital.com'\n", "'carado.moe'\n", "'nonarxiv_papers'\n", "'https://vkrakovna.wordpress.com'\n", "'https://jsteinhardt.wordpress.com'\n", "'audio-transcripts'\n", "'https://intelligence.org'\n", "'youtube'\n", "'reports'\n", "'https://aisafety.camp'\n", "'curriculum'\n", "'https://www.yudkowsky.net'\n", "'distill'\n", "```\n", "\n", "```...and this is how the arxiv papers look like:```\n", "\n", "```\n", "{\n", " \"source\": \"arxiv\", # where the dataset comes from\n", " \"source_type\": \"latex\", # the type of file the data was original in\n", " \"converted_with\": \"pandoc\", # which tool we used to convert the data in .md format\n", " \"paper_version\": paper_id,\n", " \"title\": title,\n", " \"authors\": [str(x) for x in authors], # list of authors\n", " \"date_published\": date_published,\n", " \"data_last_modified\": data_last_modified,\n", " \"url\": url,\n", " \"abstract\": abstract,\n", " \"author_comment\": author_comment,\n", " \"journal_ref\": journal_ref,\n", " \"doi\": doi,\n", " \"primary_category\": primary_category,\n", " \"categories\": categories,\n", " \"citation_level\": citation_level, # (0 = curated alignment papers, 1 = citation of curated papers, 2 = citation of citation, etc.)\n", " \"alignment_text\": is_alignment_text, # 'pos' is maunally labeled as an alignment paper, 'unlabeled' if unlabeled\n", " \"confidence_score\": confidence_scores, # this is a confidence score obtained by using the SPECTER model to classify papers to add to the dataset\n", " \"main_tex_filename\": \"main.tex\", # the main latex file needed to convert the paper\n", " \"text\": \"lots of text\", # this is where you will grab the text contents of each entry in the dataset (in .md format)\n", " \"bibliography_bbl\": \"string of bbl\",\n", " \"bibliography_bib\": \"string of bib\", # more common to have bib than bbl\n", "}\n", "```" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "https://aipulse.org: title links link authors author text (tags)\n", "\n", "None: title url text\n", "\n", "ebook: title book_title authors text (publication_date)\n", "\n", "https://qualiacomputing.com: title link authors author text (tags)\n", "\n", "alignment forum: title url authors text (tags)\n", "\n", "lesswrong: title authors url text (tags score date_published)\n", "\n", "manual: title authors text (date_published)\n", "\n", "arxiv: title authors url text (citation_level alignment_text confidence_score date_published)\n", "\n", "https://deepmindsafetyresearch.medium.com/: title url text\n", "\n", "waitbutwhy.com: title authors text (date_published)\n", "\n", "GitHub: book_title authors author text\n", "\n", "https://aiimpacts.org: title link authors author text (tags)\n", "\n", "arbital.com: title authors url text (date_published)\n", "\n", "carado.moe: title authors text (date_published)\n", "\n", "nonarxiv_papers: title authors doi text (date_published)\n", "\n", "https://vkrakovna.wordpress.com: title link authors author text (tags)\n", "\n", "https://jsteinhardt.wordpress.com: title link authors author text (tags)\n", "\n", "audio-transcripts: title authors text (date_published)\n", "\n", "https://intelligence.org: title link authors author text (tags)\n", "\n", "youtube: title authors url text (date_published)\n", "\n", "reports: title authors doi text (date_published)\n", "\n", "https://aisafety.camp: title link authors author text (tags)\n", "\n", "curriculum: title authors text (date_published)\n", "\n", "https://www.yudkowsky.net: title link authors author text (tags)\n", "\n", "distill: title authors doi text (date_published)" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "Useful links:\n", "\n", "- https://github.com/openai/openai-cookbook/blob/main/examples/Semantic_text_search_using_embeddings.ipynb\n", "\n", "- https://github.com/openai/openai-cookbook/blob/main/examples/Question_answering_using_embeddings.ipynb" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Imports" ] }, { "cell_type": "code", "execution_count": 143, "metadata": {}, "outputs": [], "source": [ "import jsonlines\n", "import numpy as np\n", "from typing import List, Dict, Tuple\n", "import re\n", "import time\n", "import random\n", "import pickle\n", "import openai\n", "from tenacity import (\n", " retry,\n", " stop_after_attempt,\n", " wait_random_exponential,\n", ") # for exponential backoff\n", "\n", "import config" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Constants" ] }, { "cell_type": "code", "execution_count": 119, "metadata": {}, "outputs": [], "source": [ "LEN_EMBEDDINGS = 1536\n", "PATH_TO_DATA = r\"C:\\Users\\Henri\\Documents\\GitHub\\AlignmentSearch\\src\\Embeddings Search\\data\\alignment_texts.jsonl\"\n", "PATH_TO_EMBEDDINGS = r\"C:\\Users\\Henri\\Documents\\GitHub\\AlignmentSearch\\src\\Embeddings Search\\data\\embeddings.npy\"\n", "PATH_TO_DATASET = r\"C:\\Users\\Henri\\Documents\\GitHub\\AlignmentSearch\\src\\Embeddings Search\\data\\dataset.pkl\"\n", "\n", "COMPLETIONS_MODEL = \"text-davinci-003\"\n", "EMBEDDING_MODEL = \"text-embedding-ada-002\"\n", "\n", "openai.api_key = config.OPENAI_API_KEY\n", "\n", "MAX_LEN_PROMPT = 5000" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Helpers" ] }, { "cell_type": "code", "execution_count": 9, "metadata": {}, "outputs": [], "source": [ "# FROM https://stackoverflow.com/a/31505798/16185542\n", "\n", "# -*- coding: utf-8 -*-\n", "alphabets= \"([A-Za-z])\"\n", "prefixes = \"(Mr|St|Mrs|Ms|Dr)[.]\"\n", "suffixes = \"(Inc|Ltd|Jr|Sr|Co)\"\n", "starters = \"(Mr|Mrs|Ms|Dr|Prof|Capt|Cpt|Lt|He\\s|She\\s|It\\s|They\\s|Their\\s|Our\\s|We\\s|But\\s|However\\s|That\\s|This\\s|Wherever)\"\n", "acronyms = \"([A-Z][.][A-Z][.](?:[A-Z][.])?)\"\n", "websites = \"[.](com|net|org|io|gov|edu|me)\"\n", "digits = \"([0-9])\"\n", "\n", "def split_into_sentences(text):\n", " text = \" \" + text + \" \"\n", " text = text.replace(\"\\n\",\" \")\n", " text = text.replace(\"?!\", \"?\")\n", " text = re.sub(prefixes,\"\\\\1\",text)\n", " text = re.sub(websites,\"\\\\1\",text)\n", " text = re.sub(digits + \"[.]\" + digits,\"\\\\1\\\\2\",text)\n", " if \"...\" in text: text = text.replace(\"...\",\"\")\n", " if \"Ph.D\" in text: text = text.replace(\"Ph.D.\",\"PhD\")\n", " text = re.sub(\"\\s\" + alphabets + \"[.] \",\" \\\\1 \",text)\n", " text = re.sub(acronyms+\" \"+starters,\"\\\\1 \\\\2\",text)\n", " text = re.sub(alphabets + \"[.]\" + alphabets + \"[.]\" + alphabets + \"[.]\",\"\\\\1\\\\2\\\\3\",text)\n", " text = re.sub(alphabets + \"[.]\" + alphabets + \"[.]\",\"\\\\1\\\\2\",text)\n", " text = re.sub(\" \"+suffixes+\"[.] \"+starters,\" \\\\1 \\\\2\",text)\n", " text = re.sub(\" \"+suffixes+\"[.]\",\" \\\\1\",text)\n", " text = re.sub(\" \" + alphabets + \"[.]\",\" \\\\1\",text)\n", " if \"”\" in text: text = text.replace(\".”\",\"”.\")\n", " if \"\\\"\" in text: text = text.replace(\".\\\"\",\"\\\".\")\n", " if \"!\" in text: text = text.replace(\"!\\\"\",\"\\\"!\")\n", " if \"?\" in text: text = text.replace(\"?\\\"\",\"\\\"?\")\n", " text = text.replace(\".\",\".\")\n", " text = text.replace(\"?\",\"?\")\n", " text = text.replace(\"!\",\"!\")\n", " text = text.replace(\"\",\".\")\n", "\n", " sentences = text.split(\"\")\n", " sentences = sentences[:-1]\n", " sentences = [s.strip() for s in sentences]\n", " \n", " if sentences == []:\n", " sentences = [text.strip()]\n", " return sentences" ] }, { "cell_type": "code", "execution_count": 13, "metadata": {}, "outputs": [], "source": [ "class MissingDataException(Exception):\n", " pass" ] }, { "cell_type": "code", "execution_count": 14, "metadata": {}, "outputs": [], "source": [ "class TextSplitter:\n", " def __init__(self, block_maxsize: int = 800, block_minsize: int = 500):\n", " self.block_maxsize = block_maxsize\n", " self.block_minsize = block_minsize\n", " self.blocks = []\n", " self.current_block = []\n", " self.current_block_len = 0\n", "\n", "\n", " def add_sentence_to_blocks(self, sentence):\n", " sentence_len = len(sentence)\n", " sentence_fits_in_current_block = self.current_block_len + sentence_len <= self.block_maxsize\n", " current_block_is_big_enough = self.current_block_len >= self.block_minsize\n", " sentence_fits_in_standalone_block = sentence_len <= self.block_maxsize\n", "\n", " if sentence_fits_in_current_block:\n", " self.current_block.append(sentence)\n", " self.current_block_len += sentence_len + 1 # +1 for the space\n", " return\n", " \n", " if current_block_is_big_enough and sentence_fits_in_standalone_block:\n", " self.blocks.append(\" \".join(self.current_block))\n", " self.current_block = [sentence]\n", " self.current_block_len = sentence_len + 1 # +1 for the space\n", " return\n", " \n", " #special cases:TODO refactor\n", " #case 1: current_block_len < block_minsize and current_block_len + sentence_len > block_maxsize\n", " #case 2: current_block_len > block_minsize but sentence_len > block_maxsize\n", " shorter_sentence = sentence[self.block_maxsize - self.current_block_len]\n", " self.current_block.append(shorter_sentence)\n", " self.blocks.append(\" \".join(self.current_block))\n", " self.current_block = []\n", " self.current_block_len = 0 \n", " \n", "\n", " def add_paragraph_to_blocks(self, paragraph):\n", " paragraph_len = len(paragraph)\n", " if self.current_block_len + paragraph_len > self.block_maxsize:\n", " sentences = split_into_sentences(paragraph)\n", " for sentence in sentences:\n", " self.add_sentence_to_blocks(sentence)\n", " return\n", " \n", " if self.block_minsize <= self.current_block_len + paragraph_len <= self.block_maxsize:\n", " self.current_block.append(paragraph)\n", " self.blocks.append(\"\\n\\n\".join(self.current_block))\n", " self.current_block = []\n", " self.current_block_len = 0\n", " return\n", " \n", " if self.current_block_len + paragraph_len < self.block_minsize:\n", " self.current_block.append(paragraph)\n", " self.current_block_len += paragraph_len + 2 # +2 for the \\n\\n\n", " return\n", " \n", " def add_text_to_blocks(self, text):\n", " paragraphs = text.split(\"\\n\\n\")\n", " for paragraph in paragraphs:\n", " self.add_paragraph_to_blocks(paragraph)\n", " if self.current_block != []:\n", " self.blocks.append(\"\\n\\n\".join(self.current_block))\n", "\n", "\n", " def split(self, text: str, signature: str) -> List[str]:\n", " \"\"\"Split text into multiple blocks and add signature to each block.\"\"\"\n", " # signature has the format : \"link, title, author\"\n", " self.add_text_to_blocks(text)\n", " \n", " return [f\"{block}\\n - {signature}\" for block in self.blocks]\n" ] }, { "attachments": {}, "cell_type": "markdown", "metadata": {}, "source": [ "## Dataset Class" ] }, { "cell_type": "code", "execution_count": 15, "metadata": {}, "outputs": [], "source": [ "error_count_dict = {\n", " \"Entry has no source.\": 0,\n", " \"Entry has no title.\": 0,\n", " \"Entry has no text.\": 0,\n", " \"Entry has no URL.\": 0,\n", " \"Entry has wrong citation level.\": 0\n", "}" ] }, { "cell_type": "code", "execution_count": 138, "metadata": {}, "outputs": [], "source": [ "class Dataset:\n", " def __init__(self,\n", " path: str, # Path to the dataset .jsonl file.\n", " sources: List[str] = None, # List of sources to include. If None, include all sources.\n", " rate_limit_per_minute: int = 60, # Rate limit for the OpenAI API.\n", " block_min_max_size: Tuple[int, int] = None, # Tuple of (min_block_size, max_block_size), used for the text splitter. If None, use default values.\n", " ):\n", " self.path = path\n", " self.sources = sources\n", " self.rate_limit_per_minute = rate_limit_per_minute\n", " self.delay_in_seconds = 60.0 / self.rate_limit_per_minute\n", " \n", " # Set up text splitter\n", " if block_min_max_size is None: self.block_min_max_size = (400, 600)\n", " else: self.block_min_max_size = block_min_max_size\n", " self.text_splitter = TextSplitter(block_maxsize=self.block_min_max_size[1], block_minsize=self.block_min_max_size[0])\n", " \n", " self.data: List[Tuple[str]] = [] # List of tuples, each containing the title of an article, its URL, and text. E.g.: [('title', 'url', 'text'), ...]\n", " self.embed_split: List[str] = [] # List of strings, each being a few paragraphs from a single article (not exceeding 1000 words).\n", " \n", " self.num_articles: Dict[str, int] = {} # Number of articles per source. E.g.: {'source1': 10, 'source2': 20, 'total': 30}\n", " if sources is None:\n", " self.num_articles['total'] = 0\n", " else:\n", " for source in sources: \n", " self.num_articles[source] = 0\n", " self.num_articles['total'] = 0\n", " \n", " self.total_char_count = 0\n", " self.total_word_count = 0\n", " self.total_sentence_count = 0\n", " self.total_block_count = 0\n", " \n", " def get_info_tmp(self):\n", " self.sources_so_far = []\n", " self.info_types: Dict[str, List[str]] = {}\n", " with jsonlines.open(self.path, \"r\") as reader:\n", " for entry in reader:\n", " if 'source' not in entry: entry['source'] = 'None'\n", " \n", " if entry['source'] not in self.sources_so_far:\n", " self.sources_so_far.append(entry['source'])\n", " self.info_types[entry['source']] = entry.keys()\n", " \n", " if 'tags' in entry:\n", " print(entry['tags'])\n", " \n", " \"\"\"\n", " {\n", " 'text', \n", " 'title', 'book_title', # If there is both, take title, otherwise take book_title\n", " 'author', 'authors', # If there is both, take author, otherwise take authors, otherwise take author\n", " 'citation_level', # must be 0 or 1\n", " 'date_published', 'published', # take first 10 chars of date_published, if it exists; else take first 16 chars of published, if it exists\n", " 'doi', 'link', 'links', 'url', # if link, take link; elif url, take url; elif doi, take doi\n", " 'tags'\n", " }\n", " \"\"\"\n", " \n", " def get_alignment_texts(self):\n", " with jsonlines.open(self.path, \"r\") as reader:\n", " for entry in reader:\n", " # Only get one in a thousand articles\n", " if random.randint(0, 3000) != 19: continue\n", " try:\n", " if 'source' not in entry: raise MissingDataException(\"Entry has no source.\")\n", " \n", " if self.sources is None:\n", " if entry['source'] not in self.num_articles: self.num_articles[entry['source']] = 1\n", " else: self.num_articles[entry['source']] += 1\n", " self.num_articles['total'] += 1\n", " else:\n", " if entry['source'] in self.sources:\n", " self.num_articles[entry['source']] += 1\n", " self.num_articles['total'] += 1\n", " else: continue\n", " \n", " text=title=author=citation_level=date_published=url=tags=None\n", " \n", " # Get text\n", " if 'text' in entry and entry['text'] != '': text = entry['text']\n", " else: raise MissingDataException(f\"Entry has no text.\")\n", " \n", " # Get title\n", " if 'title' in entry and 'book_title' in entry and entry['title'] != '': title = entry['title']\n", " elif 'book_title' in entry and entry['book_title'] != '': title = entry['book_title']\n", " else: title = None\n", " \n", " # Get author\n", " if 'author' in entry and 'authors' in entry and entry['author'] != '': author = entry['author']\n", " elif 'authors' in entry and entry['authors'] != '': author = entry['authors']\n", " elif 'author' in entry and entry['author'] != '': author = entry['author']\n", " else: author = None\n", " \n", " # Get citation level\n", " if 'citation_level' in entry:\n", " if entry['citation_level'] != 0: raise MissingDataException(f\"Entry has citation_level {entry['citation_level']}.\")\n", " \n", " # Get date published\n", " if 'date_published' in entry and entry['date_published'] != '': date_published = entry['date_published'][:10]\n", " elif 'published' in entry and entry['published'] != '': date_published = entry['published'][:16]\n", " else: date_published = None\n", " \n", " # Get URL\n", " if 'link' in entry and entry['link'] != '': url = entry['link']\n", " elif 'url' in entry and entry['url'] != '': url = entry['url']\n", " elif 'doi' in entry and entry['doi'] != '': url = entry['doi']\n", " else: url = None\n", " \n", " # Get tags\n", " if 'tags' in entry and entry['tags'] != '':\n", " if type(entry['tags']) == list: tags = ', '.join([val['term'] for val in entry['tags']])\n", " elif type(entry['tags']) == str: tags = entry['tags']\n", " else: tags = None\n", " \n", " signature = \"\"\n", " if title: signature += f\"Title: {title}, \"\n", " if author: signature += f\"Author: {author}, \"\n", " if date_published: signature += f\"Date published: {date_published}, \"\n", " if url: signature += f\"URL: {url}, \"\n", " if tags: signature += f\"Tags: {tags}, \"\n", " signature = signature[:-2]\n", "\n", " self.data.append((title, author, date_published, url, tags, text))\n", " \n", " blocks = self.text_splitter.split(text, signature)\n", " self.embed_split.extend(blocks)\n", " \n", " self.total_char_count += len(entry['text'])\n", " self.total_word_count += len(entry['text'].split())\n", " self.total_sentence_count += len(split_into_sentences(entry['text']))\n", " self.total_block_count += len(blocks)\n", " \n", " except MissingDataException as e:\n", " if str(e) not in error_count_dict:\n", " error_count_dict[str(e)] = 0\n", " error_count_dict[str(e)] += 1\n", "\n", " @retry(wait=wait_random_exponential(min=1, max=100), stop=stop_after_attempt(10))\n", " def get_embedding(self, text: str, delay_in_seconds: float = 0) -> np.ndarray:\n", " time.sleep(delay_in_seconds)\n", " result = openai.Embedding.create(model=EMBEDDING_MODEL, input=text)\n", " return result[\"data\"][0][\"embedding\"]\n", "\n", " def get_embeddings(self):\n", " self.embeddings = np.array([self.get_embedding(text, delay_in_seconds=self.delay_in_seconds) for text in self.embed_split])\n", " \n", " def save_embeddings(self, path: str):\n", " np.save(path, self.embeddings)\n", " \n", " def load_embeddings(self, path: str):\n", " self.embeddings = np.load(path)\n", " \n", " def save_class(self, path: str):\n", " with open(path, 'wb') as f:\n", " pickle.dump(self, f)" ] }, { "cell_type": "code", "execution_count": 139, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "1416\n" ] } ], "source": [ "dataset = Dataset(path=PATH_TO_DATA, sources=None)\n", "dataset.get_alignment_texts()\n", "dataset.get_embeddings()\n", "dataset.save_embeddings(PATH_TO_EMBEDDINGS)\n", "dataset.save_class(PATH_TO_DATASET)" ] }, { "cell_type": "code", "execution_count": null, "metadata": {}, "outputs": [], "source": [ "# with open(PATH_TO_DATASET, 'rb') as f:\n", "# dataset = pickle.load(f)" ] }, { "cell_type": "code", "execution_count": 157, "metadata": {}, "outputs": [], "source": [ "class SearchAndAnswer:\n", " def __init__(self,\n", " dataset: Dataset, # Dataset object containing the data.\n", " ):\n", " self.dataset = dataset\n", " \n", " @retry(wait=wait_random_exponential(min=1, max=100), stop=stop_after_attempt(10))\n", " def get_embedding(self, text: str) -> np.ndarray:\n", " result = openai.Embedding.create(model=EMBEDDING_MODEL, input=text)\n", " return result[\"data\"][0][\"embedding\"]\n", " \n", " def get_top_k(self, query: str, k: int=10) -> List[Tuple[str, str, str]]:\n", " # Receives a query (str) and returns the top k articles (List[Tuple[str, str, str]]) that are most similar to the query.\n", " # Each tuple contains the title of an article, its URL, and text.\n", " query_embedding = self.get_embedding(query)\n", " similarities = np.dot(self.dataset.embeddings, query_embedding)\n", " print(similarities.shape)\n", " top_k_indices = np.argsort(similarities)[::-1][:k]\n", " print(top_k_indices)\n", " top_k = [self.dataset.embed_split[i] for i in top_k_indices]\n", " return top_k\n", " \n", " def construct_prompt(self, question: str, texts: List[Tuple[str]]) -> str:\n", " # Receives a question (str) and a list of articles (List[Tuple[str, str, str]]) and returns a prompt (str) to be used for text generation.\n", " context = \"\\n\".join(texts)[:MAX_LEN_PROMPT]\n", " header = \"\"\"Answer the question as truthfully as possible using the provided context, and if the answer is not contained within the text below, say \"I don't know.\"\\n\\nContext:\\n\"\"\"\n", " return header + \"\".join(context) + \"\\n\\n Q: \" + question + \"\\n A:\"\n", " \n", " def answer_question(self, question: str, texts: List[Tuple[str, str, str]]) -> str:\n", " # Receives a question (str) and a list of articles (List[Tuple[str, str, str]]) and returns an answer (str) to the question.\n", " prompt = self.construct_prompt(question, texts)\n", " COMPLETIONS_API_PARAMS = {\n", " \"temperature\": 0.0,\n", " \"max_tokens\": 500,\n", " \"model\": COMPLETIONS_MODEL,\n", " }\n", " answer = openai.Completion.create(prompt=prompt, **COMPLETIONS_API_PARAMS)[\"choices\"][0][\"text\"].strip(\" \\n\")\n", " return answer\n", " \n", " def search_and_answer(self, question: str, k: int=10, HyDE: bool=False) -> str:\n", " # Receives a question (str) and returns an answer (str) to the question.\n", " if HyDE:\n", " raise NotImplementedError\n", " else:\n", " top_k = self.get_top_k(question, k)\n", " answer = self.answer_question(question, top_k)\n", " return answer\n" ] }, { "cell_type": "code", "execution_count": 158, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "(1416,)\n", "[ 82 916 1171]\n", "The need for a high quality alignment dataset for very capable models.\n" ] } ], "source": [ "SA = SearchAndAnswer(dataset=dataset)\n", "prompt = \"Name a problem in AI Alignment.\"\n", "answer = SA.search_and_answer(prompt, 3, HyDE=False)\n", "print(answer)" ] }, { "cell_type": "code", "execution_count": 147, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Source Truth Empirical Difference\n", "total 41614 14 41600 \n", "lesswrong 28479 11 28468 \n", "alignment forum 2138 1 2137 \n", "arxiv 8007 2 8005 \n", "\n", " Truth Empirical Difference\n", "Word Count 53550146 23344 53526802 \n", "Character Count 351767163 143048 351624115 \n" ] } ], "source": [ "# self.path = path\n", "# self.sources = sources\n", "# self.max_data_length = max_data_length\n", "# self.len_embeddings = len_embeddings\n", "\n", "\n", "# self.data: List[Tuple[str, str, str]] = [] # List of tuples, each containing the title of an article, its URL, and text. E.g.: [('title', 'url', 'text'), ...]\n", "# self.embed_split: List[str] = [] # List of strings, each being a few paragraphs from a single article (not exceeding 1000 words).\n", "\n", "# self.num_articles: Dict[str, int] = {} # Dict of number of articles from each source, with total number of articles. Initialize num_articles to 0 for each source.\n", "# for source in sources: self.num_articles[source] = 0\n", "# self.num_articles['total'] = 0\n", "\n", "# self.total_char_count = 0\n", "# self.total_word_count = 0\n", "# self.total_sentence_count = 0\n", "# self.total_paragraph_count = 0\n", "\n", "num_articles_truth = {\n", " 'https://aipulse.org': 23,\n", " 'ebook': 23,\n", " 'https://qualiacomputing.com': 278,\n", " 'alignment forum': 2138,\n", " 'lesswrong': 28252 + 227,\n", " 'manual': \"?\",\n", " 'arxiv': 707 + 1679 + 1000 + 4621,\n", " 'https://deepmindsafetyresearch.medium.com/': 10,\n", " 'waitbutwhy.com': 2,\n", " 'GitHub': \"?\",\n", " 'https://aiimpacts.org': 227,\n", " 'arbital.com': 223,\n", " 'carado.moe': 59,\n", " 'nonarxiv_papers': \"?\",\n", " 'https://vkrakovna.wordpress.com': 43,\n", " 'https://jsteinhardt.wordpress.com': 39,\n", " 'audio-transcripts': 25 + 12,\n", " 'https://intelligence.org': 479,\n", " 'youtube': 457,\n", " 'reports': \"?\",\n", " 'https://aisafety.camp': 8,\n", " 'curriculum': \"?\",\n", " 'https://www.yudkowsky.net': 23,\n", " 'distill': 49,\n", " 'total': 2138+28252+707+1679+1000+4621+23+227+23+8+59+111+10+17+7+479+39+278+43+2+23+420+323+49+457+25+12+223+227+132 \n", "}\n", "word_count_truth = 53_550_146\n", "char_count_truth = 351_767_163\n", "\n", "# Print table. First row has Truth and Empirical findings.\n", "print(f\"{'Source':<20} {'Truth':<10} {'Empirical':<10} {'Difference':<10}\")\n", "for source in dataset.num_articles:\n", " try:\n", " print(f\"{source[:20]:<20} {num_articles_truth[source]:<10} {dataset.num_articles[source]:<10} {num_articles_truth[source] - dataset.num_articles[source]:<10}\")\n", " except TypeError:\n", " print(f\"{source[:20]:<20} {num_articles_truth[source]:<10} {dataset.num_articles[source]:<10} {'UNKNOWN':<10}\")\n", "\n", "# Compare true and empirical word counts and character counts\n", "print(f\"\\n{'':<20} {'Truth':<10} {'Empirical':<10} {'Difference':<10}\")\n", "print(f\"{'Word Count':<20} {word_count_truth:<10} {dataset.total_word_count:<10} {word_count_truth - dataset.total_word_count:<10}\")\n", "print(f\"{'Character Count':<20} {char_count_truth:<10} {dataset.total_char_count:<10} {char_count_truth - dataset.total_char_count:<10}\")" ] } ], "metadata": { "kernelspec": { "display_name": "Python 3", "language": "python", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.10.6" }, "orig_nbformat": 4, "vscode": { "interpreter": { "hash": "369f2c481f4da34e4445cda3fffd2e751bd1c4d706f27375911949ba6bb62e1c" } } }, "nbformat": 4, "nbformat_minor": 2 }