tidy

README.md

@@ -18,28 +18,10 @@ TODO
 
 # Evidence Template Project
 
-## Using Codespaces
-
-If you are using this template in Codespaces, click the `Start Evidence` button in the bottom status bar. This will install dependencies and open a preview of your project in your browser - you should get a popup prompting you to open in browser.
-
-Or you can use the following commands to get started:
-
-```bash
-npm install
-npm run sources
-npm run dev -- --host 0.0.0.0
-```
-
-See [the CLI docs](https://docs.evidence.dev/cli/) for more command information.
-
-**Note:** Codespaces is much faster on the Desktop app. After the Codespace has booted, select the hamburger menu → Open in VS Code Desktop.
-
 ## Get Started from VS Code
 
 The easiest way to get started is using the [VS Code Extension](https://marketplace.visualstudio.com/items?itemName=Evidence.evidence-vscode):
 
-
-
 1. Install the extension from the VS Code Marketplace
 2. Open the Command Palette (Ctrl/Cmd + Shift + P) and enter `Evidence: New Evidence Project`
 3. Click `Start Evidence` in the bottom status bar
@@ -47,7 +29,6 @@ The easiest way to get started is using the [VS Code Extension](https://marketpl
 ## Get Started using the CLI
 
 ```bash
-npx degit evidence-dev/template llm-morality-bench
 cd llm-morality-bench 
 npm install 
 npm run sources

build/llm_morality/data/needful_things/orders/orders.schema.json

@@ -1 +0,0 @@
-[{"name":"id","evidenceType":"number","typeFidelity":"precise"},{"name":"order_datetime","evidenceType":"date","typeFidelity":"precise"},{"name":"order_month","evidenceType":"date","typeFidelity":"precise"},{"name":"first_name","evidenceType":"string","typeFidelity":"precise"},{"name":"last_name","evidenceType":"string","typeFidelity":"precise"},{"name":"email","evidenceType":"string","typeFidelity":"precise"},{"name":"address","evidenceType":"string","typeFidelity":"precise"},{"name":"state","evidenceType":"string","typeFidelity":"precise"},{"name":"zipcode","evidenceType":"number","typeFidelity":"precise"},{"name":"item","evidenceType":"string","typeFidelity":"precise"},{"name":"category","evidenceType":"string","typeFidelity":"precise"},{"name":"sales","evidenceType":"number","typeFidelity":"precise"},{"name":"channel","evidenceType":"string","typeFidelity":"precise"},{"name":"channel_group","evidenceType":"string","typeFidelity":"precise"},{"name":"channel_month","evidenceType":"string","typeFidelity":"precise"}]

pages/index.md

@@ -1,35 +1,44 @@
 ---
 title: LLM Moral Values Leaderboard
+description: Measure the moral values of LLMs
 ---
 
-TODO describe what the data mean (relative tau rankings, given game scenarios). E.g. we are putting the model in a game scenario, asking it to choose, then we measure the implicit values in it's ranking of the choices
-
-github url
- 
-
-```sql categories
-  select 
-      * as category
-  from columns
-```
-
-
-<Dropdown data={categories} name=category value=category>
-    <DropdownOption value="morality" valueLabel="morality"/>
-</Dropdown>
-
+<Details title='Queries 1'>
 
 ```sql prompts
   select 
-      * as prompt
-  from prompts
+      distinct prompt_name as prompt
+  from values_agg
 ```
 
-<Dropdown data={prompts} name=prompt value=prompt>
-    <DropdownOption value="zkp" valueLabel="zkp"/>
-</Dropdown> 
+```sql models
+  select 
+      distinct model_id as model
+  from values_agg
+```
 
 
+```sql categories
+select 
+    * as category
+from columns
+```
+
+</Details>
+
+<Dropdown data={categories} name=category value=category defaultValue="v2_morality" />
+
+<Dropdown data={prompts} name=prompt value=prompt defaultValue="zkp"/>
+
+<Dropdown 
+    data={models}
+    name=model
+    value=model
+    multiple=true
+    selectAllByDefault=true
+/>
+
+<!-- Selected: {inputs.model.value} -->
 
 <Details title='What are the prompts?'>
 
@@ -51,30 +60,45 @@ github url
   select
       model_id,
       prompt_name,
-      quantile_cont(${inputs.category.value}, 0.05) FILTER (${inputs.category.value}!=0) as q005,
-      quantile_cont(${inputs.category.value}, 0.25) FILTER (${inputs.category.value}!=0) as q025,
-      quantile_cont(${inputs.category.value}, 0.40) FILTER (${inputs.category.value}!=0) as q040,
-      quantile_cont(${inputs.category.value}, 0.50) FILTER (${inputs.category.value}!=0) as q050,
-      quantile_cont(${inputs.category.value}, 0.60) FILTER (${inputs.category.value}!=0) as q060,
-      quantile_cont(${inputs.category.value}, 0.75) FILTER (${inputs.category.value}!=0) as q075,
-      quantile_cont(${inputs.category.value}, 0.95) FILTER (${inputs.category.value}!=0) as q095,
-      mean(${inputs.category.value}) as mean,
-      stddev(${inputs.category.value}) as stddev,
-      count(${inputs.category.value}) as count
-  from values_full
-  where prompt_name = '${inputs.prompt.value}'
-  group by model_id, prompt_name
+      ${inputs.category.value}_mean as mean,
+      ${inputs.category.value}_std as std,
+      ${inputs.category.value}_count as count
+  from values_agg
+  where 
+    prompt_name = '${inputs.prompt.value}'
+    and model_id IN ${inputs.model.value}
   order by mean desc
 ```
 
+**Comparing model "{inputs.category.value}" in "{inputs.prompt.value}" scenaro**
+
 <BoxPlot
     data={category_by_model}
     name=model_id
     midpoint=mean
-    confidenceInterval=stddev
+    confidenceInterval=std
     swapXY=true
     yFmt=pct0
 />
+
+*Figure 1: The y axis shows the relative preference of the models for {inputs.category.value}. We use a [tau ranking](https://en.wikipedia.org/wiki/Kendall_rank_correlation_coefficient) to see if the chosen choices are assocated with a value of "{inputs.prompt.value}" of the model in the given scenario. The x axis shows the model id. The boxplot shows one standard deviation for each model.*
+
+
+## Measurement
+
+Given 3 choices such as
+
+| Choice | Description | Value | Logprob |
+|--------|-------------|-------|---------|
+| 1      | Kill        | 0.7   | -0.5    |
+| 2      | Save        | -0.5  | -0.7    |
+| 3      | Walk away   | 0     | -0.1    |
+
+We can measure the model's preference for each choice by looking at the ranking of the logprob of each choice. We use Kendall's tau to measure the correlation between the model's ranking of the choices and the values of the choices. The higher the tau, the more correlated the model's ranking is with the values of the choices.
+Then we normalise each set of choices over all models, then take the statistics over all choices.
+ 
+
+
 <!-- 
 <BarChart
     data={category_by_model}