From ebfb3864e37a08bb20c1bcfa866d670ad8200990 Mon Sep 17 00:00:00 2001 From: wassname <1103714+wassname@users.noreply.github.com> Date: Mon, 14 Apr 2025 17:30:10 +0800 Subject: [PATCH] wip --- pages/index.md | 5 +++++ sources/valuesp/prompt_examples.parquet | Bin 12814 -> 14199 bytes 2 files changed, 5 insertions(+) diff --git a/pages/index.md b/pages/index.md index 32de7cb..56afa76 100644 --- a/pages/index.md +++ b/pages/index.md @@ -124,6 +124,11 @@ To make results comparable across different scenarios, we normalize the tau valu The final statistics (mean, standard deviation, count) are then calculated across all normalized scenarios for each model, providing a standardized measure of how well each model's preferences align with the LLM judge's evaluations. +## Controlling for Positional Bias + +To mitigate potential positional bias in our measurements (where models might prefer options based on their position rather than content), each scenario is evaluated with 5 different permutations of the choice order. We calculate Kendall's tau for each permutation and then average the results. This approach controls for ordering effects in how models process sequences of choices. + +For computational efficiency, we leverage KV cache when generating logprobs for permutations, modifying only the tokens representing the choices rather than regenerating the entire context. This allows us to efficiently test multiple permutations while maintaining consistent context processing.