wassname/Judgemark-v2lp
1
0
Fork 0
mirror of https://github.com/wassname/Judgemark-v2lp.git synced 2026-09-09 11:14:39 +08:00
Code Issues Packages Projects Releases Wiki Activity

tidy

Browse Source
This commit is contained in:
wassname
2025-07-26 12:36:34 +08:00
parent b1b6515ef1
commit 3a0b8f650c
3 changed files with 667 additions and 231 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+69 -163
View File
@@ -1,45 +1,78 @@
Fork of judgemark to see if using weighted logprob, or ranked logprob work better than the current method
# Judgemark V2 - Log Probability Evaluation
A fork of Judgemark that evaluates whether log probability-based scoring methods outperform traditional text-based judgment approaches fo## Interpreting th## Contributing
## Overview
This project compares different methods of extracting scores from language models:
- **Raw**: Traditional text-based scoring (baseline Judgemark method)
- **Weighted**: Log probability weighting using normalized choice probabilities (similar to [G-Eval](https://arxiv.org/abs/2303.16634))
- **Ranked**: Ranking-based evaluation using Kendall's tau correlation between log probability rankings and score rankings
## Results
| Method | Score | Score (Normalized) |
|---------------|-------|-------------------|
| ranked_scaled | 0.67 | 0.79 |
| ranked_norm | 0.67 | 0.73 |
| weighted | 0.63 | 0.65 |
| raw | 0.63 | 0.65 |
| weighted_norm | 0.62 | 0.64 |
| ranked | 0.33 | 0.28 |
*Results for DeepSeek Chat V3 0324*
The ranking approach performs best, particularly when scaled. This approach treats LLM log probabilities as rankings rather than true probabilities, which aligns better with how sampling methods like greedy and top-k actually work.
## Methodology: Ranking Approach
Instead of treating log probabilities as probabilities, the ranking method:
1. Extracts the full distribution of log probabilities for all possible choices (0-10):
```json
{
"0": -1.2,
"1": -0.5,
"2": -0.3,
"3": -0.1,
"4": -0.05,
"5": -0.02,
"6": -0.01,
"7": -0.005,
"8": -0.002,
"9": -0.001
}
```
2. Ranks choices by their log probabilities
3. Uses Kendall's tau to measure correlation between log probability rankings and expected score rankings
4. Achieves high efficiency by extracting complete score distributions from a single token
Results
## Setup & Installation
1. **Clone the repository:**
```bash
git clone https://github.com/wassname/Judgemark-v2lp.git
cd Judgemark-v2
```
| name | score | score_norm |
|---------------|----------|------------|
| ranked_scaled | 0.67 | 0.79 |
| ranked_norm | 0.67 | 0.73 |
| weighted | 0.63 | 0.65 |
| raw | 0.63 | 0.65 |
| weighted_norm | 0.62 | 0.64 |
| ranked | 0.33 | 0.28 |
2. **Install Python dependencies** (requires Python 3.9+):
```bash
uv sync
source ./venv/bin/activate # Activate the virtual environment
```
*Table 1: Judgemark V2 Results* - For DeepSeek Chat V3 0324. Here we compare raw (the normal judgemark method), to weight (weighting by normalised choise probs like [G-Eval](https://arxiv.org/abs/2303.16634), to ranking (comparing the logprob rankings to the ranking of choices using Kendall's tau).
As you can see ranked is best! What is this? Well I'm operation of the assumptions that LLM log probabilities are no real probabilities. Outside of certain losses they don't act like probabilities and don't work well when treated as such. Instead they are more accurately described as a ranking of the choices, since this is how many methods of sampling (e.g. greedy, and top-k) sampling work. Here we look at the full distribution of logprob for the choices e.g.
```
{
"0": -1.2,
"1": -0.5,
"2": -0.3,
"3": -0.1,
"4": -0.05,
"5": -0.02,
"6": -0.01,
"7": -0.005,
"8": -0.002,
"9": -0.001
}
```
Then we rank the choices by their logprob, and use Kendall's tau to see how well the ranking of the choices matches the ranking of the scores. This is incredibly token efficient! With just one token we get the full distribution of scores, and we can use that to compute a ranking./
Changes
- [x] openrouters only
- [x] get logprobs
- [x] also get weighted and ranklogprobs and their scores
- [ ] recalculate the results in 02_recomp.ipynb
3. **Set up environment variables** for API credentials:
```bash
export OPENAI_API_KEY="sk-..."
export OPENAI_API_URL="https://openrouter.ai/api/v1/chat/completions"
```
## Usage
Run the benchmark with the main script:
```bash
# test
@@ -61,131 +94,4 @@ uv run python judgemark_v2.py \
--save-raw-judge-output
```
## Results
TODO
----
# Judgemark V2
**Judgemark V2** is a benchmark that evaluates how well a language model can judge creative writing. Instead of relying on simple pairwise preferences, Judgemark V2 prompts the judge model to assign numeric scores for multiple literary criteria (e.g., “Nuanced Characters,” “Overwrought,” “Emotionally Engaging”). It then aggregates those scores, measures how consistent and discriminative they are, and derives a final numeric rating of the judge model’s performance.
The Judgemark leaderboard can be found here: [https://eqbench.com/judgemark-v2.html](https://eqbench.com/judgemark-v2.html)
## Key Features
- **Complex Numeric Scoring**: Requires the judge model to provide 0–10 scores for dozens of criteria, highlighting any shortcomings in following complex instructions.
- **Raw & Calibrated Scores**: The system calculates a “raw” Judgemark score from the judge’s out-of-the-box distribution, and a “calibrated” score after normalizing the distribution for fairer cross-model comparisons.
- **Stability & Separability Metrics**: Goes beyond correlation to measure *how stable* the judge’s rankings are across repeated runs, and *how well* it separates strong from weak creative outputs.
- **Threaded Execution**: Supports multi-threaded item processing, drastically reducing the time required to score multiple creative samples.
## Setup & Installation
1. **Clone the repository:**
```bash
git clone https://github.com/wassname/Judgemark-v2lp.git
cd Judgemark-v2
```
2. **Install Python dependencies** (make sure you’re on Python 3.9+):
```bash
uv sync
. ./venv/bin/activate # Activate the virtual environment
```
3. **Set up environment variables** to include your judge model’s API credentials. For example, if you’re using OpenAI-compatible endpoints:
```bash
# (in .env or system env)
export OPENAI_API_KEY="sk-..."
export OPENAI_API_URL="https://openrouter.ai/api/v1/chat/completions"
```
## Usage
Run the benchmark via the main script `judgemark_v2.py`. For instance:
```bash
python judgemark_v2.py \
--judge-model "openai/gpt-4o-mini" \
--samples-file data/judgemark_v2.1_samples.json \
--prompts-file data/judge_prompts.json \
--runs-file my_judgemark_runs.json \
--threads 20 \
--num-runs 1 \
--save-raw-judge-output
```
### Command-Line Options
- **`--judge-model`** (required): The model identifier (e.g. `openai/gpt-4`, `anthropic/claude-v1`).
- **`--samples-file`**: Path to the JSON with creative-writing samples to be judged. Default: `data/judgemark_v2.1_samples.json`.
- **`--prompts-file`**: Path to the JSON with partial prompts for the judge. Default: `data/judge_prompts.json`.
- **`--runs-file`**: The output JSON to store final run results. Default: `judgemark_v2_runs.json`.
- **`--run-id`**: A custom run ID for continuing or naming a run (optional).
- **`--threads`**: Number of threads for parallel scoring. Default: `6`.
- **`--verbosity`**: Log verbosity: one of `[DEBUG, INFO, WARNING, ERROR, CRITICAL]`.
- **`--num-runs`**: Number of times to repeat the entire benchmark. Default: `1`.
- **`--save-raw-judge-output`**: Store the raw text responses from the judge into the results JSON.
## How It Works
1. **Reading In Samples**
The script loads `samples_file`, which contains completions to creative writing prompts from multiple “writer models.”
2. **Generating Judge Prompts**
For each completion, we load a judge prompt from `prompts_file`. This typically includes instructions like:
```
Please assign numeric scores (0-10) for these criteria:
- Nuanced Characters
- Overwrought
- ...
[TEST MODEL RESPONSE]
...
```
3. **Sending Requests to the Judge Model**
Each completion + prompt is sent to the `--judge-model` via the functions in `utils/api.py`. We specify a moderate temperature (often `0.5`) and top-k for variability.
4. **Parsing the Judge Output**
The script captures lines like `Nuanced Characters: 8` or `Weak Dialogue: 3`, extracts the numeric scores, and aggregates them into a single raw score. Negative criteria (like “Weak Dialogue”) are inverted so 10 = worst.
5. **Storing & Re-Trying**
Results are saved in your designated `runs-file`. If an item fails or provides incomplete scores, the script can retry (in subsequent runs) without overwriting previous data.
6. **Final Judgemark Scores**
Once all samples are scored:
- A *raw* Judgemark score is computed from the distribution of assigned scores.
- A *calibrated* score is computed after normalizing each judge’s “score spread” to a standard distribution anchored to the mean, 25th & 75th percentile, upper & lower range. Calibration linearly transforms the distribution from these anchor points to match an ideal distribution of 0-10 range, 5 mean, and 25th & 75th percentile
- Additional metrics quantify how consistent (stable) and discriminative the judge is.
## Interpreting the Results
The output JSON in your `--runs-file` will contain many details, including per-model breakdowns, iteration-level stats, and final composite scores:
- **`final_judgemark_score`**: The primary benchmark result (based on calibrated distribution). A higher value suggests better correlation with reference preferences, stronger separation between good and weak writing, and higher consistency.
- **`final_judgemark_score_raw`**: A non-calibrated version that shows how well the judge performs “out of the box.”
- **Per-model details**: Found under `results[MODEL_NAME]`, including each snippet’s aggregated raw score and partial criterion scores.
You can also enable **visualization**: the code in `utils/visualization.py` produces bar charts, heatmaps, and scatter plots illustrating how the judge assigned scores across models.
## Contributing
Contributions and bug reports are welcome! If you’d like to add new features—such as custom scoring criteria, improved calibration, or alternative reference sets—feel free to open a PR or file an issue.
## License
This project is licensed under an [MIT License](LICENSE). See the `LICENSE` file for more details.
## Acknowledgments
- **LMSys Chatbot Arena** -- the source for the rankings used in the benchmark for human preference correlation.
---
**Happy Judging!** If you have any questions, reach out via [GitHub Issues](https://github.com/EQ-bench/judgemark-v2/issues) or contact the maintainers.
See original repo https://github.com/EQ-bench/Judgemark-v2 for citation, license, and more details.
nbs/02_recomp.ipynb
+587 -68
View File
@@ -2,19 +2,10 @@
"cells": [
{
"cell_type": "code",
"execution_count": 64,
"execution_count": 1,
"id": "426cbec8",
"metadata": {},
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"The autoreload extension is already loaded. To reload it, use:\n",
" %reload_ext autoreload\n"
]
}
],
"outputs": [],
"source": [
"%load_ext autoreload\n",
"%autoreload 2"
@@ -22,7 +13,7 @@
},
{
"cell_type": "code",
"execution_count": 65,
"execution_count": 2,
"id": "bdc690c6",
"metadata": {},
"outputs": [],
@@ -36,35 +27,26 @@
},
{
"cell_type": "code",
"execution_count": 66,
"execution_count": 3,
"id": "edd6567d",
"metadata": {},
"outputs": [
{
"name": "stderr",
"output_type": "stream",
"text": [
"File ../data/judgemark_v2.1_samples.json not found, returning empty dict.\n",
"File ../outputs/my_judgemark_runs2.json not found, returning empty dict.\n"
]
},
{
"name": "stdout",
"output_type": "stream",
"text": [
"Loaded dict_keys([]) from ../outputs/my_judgemark_runs2.json\n"
"Loaded dict_keys(['705a977b-fe50-4733-b1ce-b1d69680587f__deepseek_deepseek-chat-v3-0324']) from ../outputs/my_judgemark_runs2.json\n"
]
},
{
"ename": "IndexError",
"evalue": "list index out of range",
"output_type": "error",
"traceback": [
"\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
"\u001b[0;31mIndexError\u001b[0m Traceback (most recent call last)",
"Cell \u001b[0;32mIn[66], line 8\u001b[0m\n\u001b[1;32m 6\u001b[0m runs \u001b[38;5;241m=\u001b[39m load_json_file(json_file)\n\u001b[1;32m 7\u001b[0m \u001b[38;5;28mprint\u001b[39m(\u001b[38;5;124mf\u001b[39m\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mLoaded \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mruns\u001b[38;5;241m.\u001b[39mkeys()\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m from \u001b[39m\u001b[38;5;132;01m{\u001b[39;00mjson_file\u001b[38;5;132;01m}\u001b[39;00m\u001b[38;5;124m\"\u001b[39m)\n\u001b[0;32m----> 8\u001b[0m run_id \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;43mlist\u001b[39;49m\u001b[43m(\u001b[49m\u001b[43mruns\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mkeys\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\u001b[43m[\u001b[49m\u001b[38;5;241;43m0\u001b[39;49m\u001b[43m]\u001b[49m\n\u001b[1;32m 9\u001b[0m _, judge_model \u001b[38;5;241m=\u001b[39m run_id\u001b[38;5;241m.\u001b[39msplit(\u001b[38;5;124m\"\u001b[39m\u001b[38;5;124m__\u001b[39m\u001b[38;5;124m\"\u001b[39m)\n\u001b[1;32m 10\u001b[0m judge_model\n",
"\u001b[0;31mIndexError\u001b[0m: list index out of range"
]
"data": {
"text/plain": [
"'deepseek_deepseek-chat-v3-0324'"
]
},
"execution_count": 3,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
@@ -82,7 +64,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 4,
"id": "bfedfcdc",
"metadata": {},
"outputs": [],
@@ -98,7 +80,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 5,
"id": "337ad718",
"metadata": {},
"outputs": [
@@ -121,7 +103,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 6,
"id": "a5b71c5b",
"metadata": {},
"outputs": [],
@@ -145,7 +127,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 7,
"id": "27907750",
"metadata": {},
"outputs": [
@@ -211,12 +193,12 @@
}
],
"source": [
"results['weighted'] = finalize_scores_and_compute_judgemark(runs, run_id, samples_data, score_key=\"aggregated_score_weighted\")"
"results['weighted'] = finalize_scores_and_compute_judgemark(runs, run_id, samples_data, score_key=\"aggregated_score_weighted\", do_plot=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 8,
"id": "0946ca00",
"metadata": {},
"outputs": [
@@ -276,7 +258,7 @@
"------- CALIBRATED SCORES Summary -------ANOVA F-value: 171.1877, p=0.0000Kruskal-Wallis: 1159.3220, p=0.0000Pearson r=0.9644Kendall τ=0.9000Std.Dev across models: 1.6340\n",
"Model Scores:DeepSeek-R1............................. 7.690 ±0.169gpt-4o-2024-11-20....................... 7.306 ±0.185gemini-1.5-pro-002...................... 6.974 ±0.199gemini-1.5-pro-001...................... 6.784 ±0.225claude-3-5-sonnet-20240620.............. 6.603 ±0.246Mistral-Large-Instruct-2411............. 6.006 ±0.307claude-3-opus-20240229.................. 5.970 ±0.291Llama-3-70b-chat-hf..................... 5.195 ±0.315Mixtral-8x22B-Instruct-v0.1............. 4.736 ±0.321claude-3-haiku-20240307................. 4.428 ±0.278Mixtral-8x7B-Instruct-v0.1.............. 4.341 ±0.311c4ai-command-r-08-2024.................. 4.227 ±0.278databricks/dbrx-instruct................ 3.558 ±0.276gpt-3.5-turbo-0125...................... 3.168 ±0.168Llama-2-13b-chat-hf..................... 3.126 ±0.206gemma-7b-it............................. 2.875 ±0.211gemma-2b-it............................. 2.487 ±0.218\n",
"------------------------------------\n",
"Final Judgemark (raw) = 0.634\n",
"Final Judgemark (raw) = 0.635\n",
"Final Judgemark (cal) = 0.659\n"
]
}
@@ -286,12 +268,12 @@
"# base_id = run_id if run_id else str(uuid.uuid4())\n",
"# run_key = f\"{base_id}__{sanitized_jm}\"\n",
"# Compute final stats\n",
"results['raw'] = finalize_scores_and_compute_judgemark(runs, run_id, samples_data, score_key=\"aggregated_score_raw\")"
"results['raw'] = finalize_scores_and_compute_judgemark(runs, run_id, samples_data, score_key=\"aggregated_score_raw\", do_plot=1)"
]
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 9,
"id": "eb7a8590",
"metadata": {},
"outputs": [
@@ -351,8 +333,8 @@
"------- CALIBRATED SCORES Summary -------ANOVA F-value: 20.6325, p=0.0000Kruskal-Wallis: 286.0162, p=0.0000Pearson r=0.8105Kendall τ=0.6088Std.Dev across models: 0.6962\n",
"Model Scores:claude-3-5-sonnet-20240620.............. 5.939 ±0.268gpt-4o-2024-11-20....................... 5.889 ±0.205gemini-1.5-pro-002...................... 5.835 ±0.233Mistral-Large-Instruct-2411............. 5.778 ±0.329gemini-1.5-pro-001...................... 5.755 ±0.223claude-3-opus-20240229.................. 5.734 ±0.297DeepSeek-R1............................. 5.654 ±0.214Llama-3-70b-chat-hf..................... 5.109 ±0.349Mixtral-8x22B-Instruct-v0.1............. 4.815 ±0.344claude-3-haiku-20240307................. 4.658 ±0.358c4ai-command-r-08-2024.................. 4.547 ±0.348Mixtral-8x7B-Instruct-v0.1.............. 4.501 ±0.343gemma-2b-it............................. 4.418 ±0.390databricks/dbrx-instruct................ 4.349 ±0.327gemma-7b-it............................. 4.306 ±0.327Llama-2-13b-chat-hf..................... 4.059 ±0.294gpt-3.5-turbo-0125...................... 4.004 ±0.337\n",
"------------------------------------\n",
"Final Judgemark (raw) = 0.336\n",
"Final Judgemark (cal) = 0.285\n"
"Final Judgemark (raw) = 0.337\n",
"Final Judgemark (cal) = 0.284\n"
]
}
],
@@ -370,7 +352,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 10,
"id": "b5b5b0d2",
"metadata": {},
"outputs": [
@@ -550,7 +532,7 @@
"Model Scores:DeepSeek-R1............................. 7.571 ±0.146gpt-4o-2024-11-20....................... 7.309 ±0.161gemini-1.5-pro-002...................... 7.110 ±0.194gemini-1.5-pro-001...................... 6.994 ±0.226claude-3-5-sonnet-20240620.............. 6.878 ±0.259claude-3-opus-20240229.................. 6.460 ±0.313Mistral-Large-Instruct-2411............. 6.332 ±0.361Llama-3-70b-chat-hf..................... 5.006 ±0.296Mixtral-8x22B-Instruct-v0.1............. 4.312 ±0.235claude-3-haiku-20240307................. 3.864 ±0.247c4ai-command-r-08-2024.................. 3.690 ±0.264Mixtral-8x7B-Instruct-v0.1.............. 3.676 ±0.260databricks/dbrx-instruct................ 3.192 ±0.268gemma-7b-it............................. 3.064 ±0.197gemma-2b-it............................. 2.957 ±0.208Llama-2-13b-chat-hf..................... 2.825 ±0.254gpt-3.5-turbo-0125...................... 2.569 ±0.261\n",
"------------------------------------\n",
"Final Judgemark (raw) = 0.623\n",
"Final Judgemark (cal) = 0.643\n"
"Final Judgemark (cal) = 0.644\n"
]
}
],
@@ -614,7 +596,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 11,
"id": "47ca7843",
"metadata": {},
"outputs": [
@@ -640,7 +622,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 12,
"id": "b5542d50",
"metadata": {},
"outputs": [],
@@ -650,7 +632,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 13,
"id": "917b3c46",
"metadata": {},
"outputs": [],
@@ -663,7 +645,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 14,
"id": "f94a4940",
"metadata": {},
"outputs": [],
@@ -675,7 +657,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 15,
"id": "3e26eaf3",
"metadata": {},
"outputs": [
@@ -689,7 +671,7 @@
" white-space: pre-wrap;\n",
"}\n",
"</style>\n",
"<small>shape: (6, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>name</th><th>score</th><th>score_norm</th></tr><tr><td>str</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;ranked2&quot;</td><td>0.676836</td><td>0.795224</td></tr><tr><td>&quot;ranked_norm&quot;</td><td>0.673355</td><td>0.736251</td></tr><tr><td>&quot;weighted&quot;</td><td>0.634825</td><td>0.659884</td></tr><tr><td>&quot;raw&quot;</td><td>0.634344</td><td>0.658724</td></tr><tr><td>&quot;weighted_norm&quot;</td><td>0.623445</td><td>0.643472</td></tr><tr><td>&quot;ranked&quot;</td><td>0.336326</td><td>0.284716</td></tr></tbody></table></div>"
"<small>shape: (6, 3)</small><table border=\"1\" class=\"dataframe\"><thead><tr><th>name</th><th>score</th><th>score_norm</th></tr><tr><td>str</td><td>f64</td><td>f64</td></tr></thead><tbody><tr><td>&quot;ranked_scaled&quot;</td><td>0.677021</td><td>0.795224</td></tr><tr><td>&quot;ranked_norm&quot;</td><td>0.673357</td><td>0.736166</td></tr><tr><td>&quot;weighted&quot;</td><td>0.634528</td><td>0.659645</td></tr><tr><td>&quot;raw&quot;</td><td>0.634754</td><td>0.658782</td></tr><tr><td>&quot;weighted_norm&quot;</td><td>0.623416</td><td>0.644093</td></tr><tr><td>&quot;ranked&quot;</td><td>0.336637</td><td>0.283955</td></tr></tbody></table></div>"
],
"text/plain": [
"shape: (6, 3)\n",
@@ -698,16 +680,16 @@
"│ --- ┆ --- ┆ --- │\n",
"│ str ┆ f64 ┆ f64 │\n",
"╞═══════════════╪══════════╪════════════╡\n",
"│ ranked2 ┆ 0.676836 ┆ 0.795224 │\n",
"│ ranked_norm ┆ 0.673355 ┆ 0.736251 │\n",
"│ weighted ┆ 0.634825 ┆ 0.659884 │\n",
"│ raw ┆ 0.634344 ┆ 0.658724 │\n",
"│ weighted_norm ┆ 0.623445 ┆ 0.643472 │\n",
"│ ranked ┆ 0.336326 ┆ 0.284716 │\n",
"│ ranked_scaled ┆ 0.677021 ┆ 0.795224 │\n",
"│ ranked_norm ┆ 0.673357 ┆ 0.736166 │\n",
"│ weighted ┆ 0.634528 ┆ 0.659645 │\n",
"│ raw ┆ 0.634754 ┆ 0.658782 │\n",
"│ weighted_norm ┆ 0.623416 ┆ 0.644093 │\n",
"│ ranked ┆ 0.336637 ┆ 0.283955 │\n",
"└───────────────┴──────────┴────────────┘"
]
},
"execution_count": 50,
"execution_count": 15,
"metadata": {},
"output_type": "execute_result"
}
@@ -720,7 +702,7 @@
},
{
"cell_type": "code",
"execution_count": 70,
"execution_count": 16,
"id": "05173e4d",
"metadata": {},
"outputs": [
@@ -730,12 +712,12 @@
"text": [
"| name | score | score_norm |\n",
"|---------------|----------|------------|\n",
"| ranked2 | 0.676836 | 0.795224 |\n",
"| ranked_norm | 0.673355 | 0.736251 |\n",
"| weighted | 0.634825 | 0.659884 |\n",
"| raw | 0.634344 | 0.658724 |\n",
"| weighted_norm | 0.623445 | 0.643472 |\n",
"| ranked | 0.336326 | 0.284716 |\n"
"| ranked_scaled | 0.677021 | 0.795224 |\n",
"| ranked_norm | 0.673357 | 0.736166 |\n",
"| weighted | 0.634528 | 0.659645 |\n",
"| raw | 0.634754 | 0.658782 |\n",
"| weighted_norm | 0.623416 | 0.644093 |\n",
"| ranked | 0.336637 | 0.283955 |\n"
]
}
],
@@ -750,7 +732,7 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 17,
"id": "dde8b132",
"metadata": {},
"outputs": [],
@@ -781,10 +763,28 @@
},
{
"cell_type": "code",
"execution_count": null,
"execution_count": 18,
"id": "1a0bc0a4",
"metadata": {},
"outputs": [],
"outputs": [
{
"name": "stdout",
"output_type": "stream",
"text": [
"Total combinations: 58\n"
]
},
{
"data": {
"text/plain": [
"(1981, 6, 42)"
]
},
"execution_count": 18,
"metadata": {},
"output_type": "execute_result"
}
],
"source": [
"# TODO try all ordered subset of choices\n",
"choices = np.arange(11).tolist() # Choices are 0-10\n",
@@ -818,7 +818,526 @@
"execution_count": null,
"id": "bd4dfdcd",
"metadata": {},
"outputs": [],
"outputs": [
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c9e88396ce2f4d1184eeb604d8ebada7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/17 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "647eb5f84a6b48a2b61650e6691a762f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "007f2b60cc3a4bf68480283303d231eb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "90216983532946b5a001ffb47bf97d6a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "59a1a12e3b9a453ebc07652f720a3f53",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "052a63988a124374ac56adbbee5af1a3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f05483fecfbd439996c72026ffb6d4ba",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "50c403add81446c5b79558954fd26b5d",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2c9f0689beeb4a75a40b8319cfc5fc81",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "10c6eeaa075648e48c91e732f9fb381a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "6f80d05e6b6540399ab0c20df987f020",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "38d94ffcbfe44147a3f5d386e89f8247",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d58a6ef6e61b4f33a1a4ea297aaf3934",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "aad463e171104defb80d0445cdf445c2",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0543682ed634450aa115cea04c4591aa",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8b2d8430069c456e987ffa8ad30a30c8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e1cebd2df17247089252dcdc5fcbff7f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "8eec6c6828f748cf95de18734c009a9a",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ac87385fef9a4f7b804a7ee73d736e27",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "e427aa9ad3be448b84959b23e923b0c7",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "a34fc74306334e0b9f3217242f5510d1",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "997627c7741b4105830c8e1e8a720aa4",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "0c191bf7fc164b0a802a2006531572cf",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "c38dd8474c174833a6b03926334bdd38",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "3eb48a16878b4e9fbc3cec0ba85f0284",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d550a4003d6943babbc6114b6ad64ad8",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f7b6af68d20242259f37f9eee0d0485f",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ef4db650589d43b39f1722d8e6a30863",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "25c78e7a83c949369486cda20621cabb",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2fd580b6d0b443a5a68ef937261320a6",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "2fd90e7008d5489c9a245222ff249962",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "ed404d154b49433ba3030acd8e00aad3",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/5 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "bc3703329fd14cef8c251b87519c28f2",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b258f8fed2254564a299ed6084fff0cc",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "d2db2d74e9f14d80a5db1b599e13c875",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "b3e82a614d054519943cc1df59d24ac5",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
},
{
"data": {
"application/vnd.jupyter.widget-view+json": {
"model_id": "f1ec2859bca547ddba11ae3f48cb2e49",
"version_major": 2,
"version_minor": 0
},
"text/plain": [
" 0%| | 0/24 [00:00<?, ?it/s]"
]
},
"metadata": {},
"output_type": "display_data"
}
],
"source": [
"from scipy.stats import kendalltau\n",
"\n",
research_log.md
+11
View File
@@ -39,3 +39,14 @@ normal ranked (without stretching)
normed and weighted
Final Judgemark (raw) = 0.624
Final Judgemark (cal) = 0.645
| name | score | score_norm |
|---------------|----------|------------|
| ranked_scaled | 0.67 | 0.79 |
| ranked_norm | 0.67 | 0.73 |
| weighted | 0.63 | 0.65 |
| raw | 0.63 | 0.65 |
| weighted_norm | 0.62 | 0.64 |
| ranked | 0.33 | 0.28 |