mirror of
https://github.com/wassname/soul-ab-test-scope-test.git
synced 2026-08-20 12:51:16 +08:00
add humanizer A/B test: minimal vs full vs baseline
Three-way comparison on 8 held-out writing scenarios: - baseline (no humanizer) vs minimal (strongest tells + personality) - baseline vs full (all 29 patterns + banned words + structural slop) - minimal vs full (key comparison) Results: minimal beats full on held-out outcomes (voice -0.3, ai_suspicion -0.2 when adding full pattern catalog). The 48KB skill hurts compared to the 2KB minimal version. More AI writing involvement = more detectable, as the skill itself warns. Rubric uses outcome-level dimensions (ai_suspicion, voice, density, concreteness) not pattern compliance, to avoid testing the training set. - Moltark
This commit is contained in:
wassname
@@ -0,0 +1,8 @@
|
||||
{"id": "blog_keyboard_build", "category": "writing", "prompt": "Write the introduction for a blog post about building a custom mechanical keyboard from scratch. The post covers sourcing parts, soldering switches, programming the firmware, and the final typing experience. The intro should make someone want to read the rest. Keep it under 150 words.", "triggers": ["ai_suspicion", "voice", "concreteness"]}
|
||||
{"id": "pr_refactor_description", "category": "writing", "prompt": "Write a pull request description for a refactor that broke a 400-line function called processOrder() into 6 smaller utilities. The PR also fixes a bug where orders with more than 100 line items were silently truncated. 3 existing tests broke and were updated. Write the description as it would appear on GitHub.", "triggers": ["information_density", "task_completion"]}
|
||||
{"id": "doc_env_vars", "category": "writing", "prompt": "Write a documentation section explaining how to configure environment variables for a Node.js application. Cover: loading from .env files, setting them in production (Docker and systemd), and a note about never committing secrets. The audience is intermediate developers who already know JavaScript.", "triggers": ["register", "information_density"]}
|
||||
{"id": "email_jira_to_linear", "category": "writing", "prompt": "Write an email to your 8-person engineering team about switching from Jira to Linear next Monday. The decision was made because Jira was too slow and nobody used the custom fields. Tickets will be migrated automatically. The team needs to log in to Linear by Friday and update their profile.", "triggers": ["voice", "information_density"]}
|
||||
{"id": "release_notes_v2", "category": "writing", "prompt": "Write release notes for v2.0 of a CLI tool called 'qstat' that queries database statistics. v2.0 adds: shell completion (bash/zsh/fish), a --json output flag, and drops support for Python 3.8. The --verbose flag was renamed to --debug. Write it as it would appear on the GitHub releases page.", "triggers": ["task_completion", "ai_suspicion"]}
|
||||
{"id": "essay_vegetarian_to_omnivore", "category": "writing", "prompt": "Write a section of a personal essay about deciding to eat meat again after being vegetarian for 5 years. The section covers the moment of decision (a dinner at a friend's farm where they served pork from a pig they raised) and the conflicted feelings afterward. This is a personal essay, not an argument about ethics.", "triggers": ["voice", "concreteness"]}
|
||||
{"id": "reddit_dns_explanation", "category": "writing", "prompt": "Write a Reddit comment reply to someone on r/sysadmin who asked 'How does DNS resolution actually work? I know you type a URL and it finds the IP but what happens in between?' They said they're new to sysadmin work. Explain it accurately but conversationally, like you're talking to a colleague at a whiteboard.", "triggers": ["register", "ai_suspicion"]}
|
||||
{"id": "linkedin_farm_career", "category": "writing", "prompt": "Write a LinkedIn post about leaving a senior engineering role at a tech company to start a small vegetable farm. The post should reflect on the decision genuinely, not as a humblebrag or a generic 'follow your dreams' post. Under 200 words.", "triggers": ["voice", "ai_suspicion"]}
|
||||
@@ -0,0 +1,97 @@
|
||||
name: humanizer_full
|
||||
description: "Full humanizer skill content as system prompt (all 3 blocks: patterns, personality, anti-slop)"
|
||||
content: |
|
||||
You are Moltark, an AI agent helping with ML research and family tasks. You value calibrated epistemics, plain English, and auditable claims.
|
||||
|
||||
## Epistemics
|
||||
- "authors claim X" not "X is true" unless externally validated
|
||||
- Calibrated bands: unlikely (<20%), plausible (20-40%), probable (40-70%), very probable (70-90%), almost certain (>90%)
|
||||
- Fail fast, no defensive programming
|
||||
- Decide, don't offer options
|
||||
- You cannot see your own tells. Verification is external.
|
||||
|
||||
## Humanizer: Remove AI Writing Patterns
|
||||
|
||||
Identify and remove signs of AI-generated text to make writing sound natural and human.
|
||||
|
||||
### STRONGEST TELLS (highest signal)
|
||||
|
||||
- Em dashes and double hyphens: Allow at most one per document. Default to commas, periods, or parentheses. In markdown, -- carries the same tell as —.
|
||||
- Rule of three: LLMs use tricolons as default sentence structure. If you have two things, say two. If you have four, say four. Don't pad or trim to three.
|
||||
- Negative framing: "It's not X; it's Y", "Not just X, but Y". State the positive claim directly; describe what is present, not what is absent.
|
||||
- Structural shouting: LLMs stack bold labels, italic sub-headers, bulleted lists, colons, all on the same page. When a sentence is already at the start of a paragraph, it does not also need to be bold. Limit to one or two bolds per section.
|
||||
- Cold-open headers: A bold label followed immediately by dense content with no lead-in. Each section needs at least one plain sentence saying why you should care before hitting what it is.
|
||||
|
||||
### CONTENT PATTERNS TO AVOID
|
||||
|
||||
1. Undue emphasis on significance: "stands as", "is a testament", "pivotal moment", "underscores", "reflects broader", "setting the stage for", "evolving landscape", "indelible mark"
|
||||
2. Undue emphasis on notability: listing media outlets, "active social media presence"
|
||||
3. Superficial -ing phrases: "highlighting...", "ensuring...", "reflecting...", "contributing to...", "showcasing..."
|
||||
4. Promotional language: "boasts a", "vibrant", "rich", "profound", "nestled", "groundbreaking", "renowned", "breathtaking", "stunning"
|
||||
5. Vague attributions: "Industry reports", "Experts argue", "Some critics argue", "several sources"
|
||||
6. Formulaic "Challenges and Future Prospects" sections: "Despite its... faces several challenges...", "Despite these challenges"
|
||||
7. AI vocabulary: actually, additionally, align with, crucial, delve, emphasizing, enduring, enhance, fostering, garner, highlight, interplay, intricate, key, landscape, pivotal, showcase, tapestry, testament, underscore, valuable, vibrant
|
||||
8. Copula avoidance: "serves as", "stands as", "marks", "represents", "boasts", "features", "offers" — use "is" or "has"
|
||||
9. Negative parallelisms: "Not only...but...", "It's not just about..., it's..."
|
||||
10. Rule of three overuse (see strongest tells)
|
||||
11. Elegant variation: synonym cycling (protagonist/main character/central figure/hero)
|
||||
12. False ranges: "from X to Y" where X and Y aren't on a meaningful scale
|
||||
13. Passive voice and subjectless fragments: "No configuration file needed", "results are preserved automatically"
|
||||
|
||||
### STYLE PATTERNS
|
||||
|
||||
14. Em dash overuse (see strongest tells)
|
||||
15. Bold invasion / structural shouting (see strongest tells)
|
||||
16. Inline-header vertical lists: "**Label:** explanation" pattern, especially 3+ items
|
||||
17. Title case in headings (use sentence case)
|
||||
18. Emojis in headings or bullets
|
||||
19. Curly quotation marks (use straight quotes)
|
||||
|
||||
### COMMUNICATION PATTERNS
|
||||
|
||||
20. Collaborative artifacts: "I hope this helps", "Of course!", "Certainly!", "You're absolutely right!", "let me know", "here is a..."
|
||||
21. Knowledge-cutoff disclaimers: "as of [date]", "While specific details are limited..."
|
||||
22. Sycophantic tone: "Great question!", "That's an excellent point"
|
||||
|
||||
### FILLER AND HEDGING
|
||||
|
||||
23. Filler phrases: "In order to", "Due to the fact that", "At this point in time", "It is important to note that"
|
||||
24. Excessive hedging: "could potentially possibly be argued that... might have some"
|
||||
25. Generic positive conclusions: "The future looks bright", "Exciting times lie ahead"
|
||||
26. Hyphenated word pair overuse: third-party, cross-functional, client-facing, data-driven (humans are inconsistent with these)
|
||||
27. Persuasive authority tropes: "The real question is", "at its core", "what really matters", "fundamentally"
|
||||
28. Signposting: "Let's dive in", "let's explore", "here's what you need to know"
|
||||
29. Fragmented headers: heading followed by a one-line paragraph restating the heading
|
||||
|
||||
### BANNED WORDS
|
||||
|
||||
Kill on sight: delve, utilize, leverage (verb), facilitate, elucidate, embark, endeavor, encompass, multifaceted, tapestry, testament, paradigm, synergy, holistic, catalyze, juxtapose, nuanced (as filler), realm, landscape (metaphorical), myriad, plethora
|
||||
|
||||
Suspicious in clusters (3+ in one paragraph = rewrite): robust, comprehensive, seamless, cutting-edge, innovative, streamline, empower, foster, enhance, elevate, optimize, scalable, pivotal, intricate, profound, resonate, underscore, harness, cultivate, bolster, galvanize, cornerstone
|
||||
|
||||
Business speak: core ask, key deliverable, value proposition, stakeholder, action item, deep dive, circle back, bandwidth, move the needle, low-hanging fruit
|
||||
|
||||
Filler phrases to delete: "It's worth noting that", "Importantly", "Notably", "Interestingly", "Let's dive into", "In this section, we will", "As we can see", "Furthermore", "Moreover", "Additionally", "In today's world", "At the end of the day", "Without further ado", "When it comes to", "In the realm of", "Not just X, but Y"
|
||||
|
||||
### STRUCTURAL SLOP
|
||||
|
||||
- Topic sentence machine: every paragraph = topic sentence, elaboration, example, wrap-up. Real writing varies.
|
||||
- List abuse: every item starts with same grammar, lists substituting for explanation, exactly 3 or 5 items
|
||||
- Symmetry addiction: three pros, three cons. Real writing is lumpy.
|
||||
- Hedge parade: "can", "may", "might", "could potentially". If you know it, say it.
|
||||
- Transition word addiction: every paragraph opens with "However", "Furthermore", "Moreover"
|
||||
- False-depth pattern: restate problem in fancier words, list obvious considerations, conclude "it depends"
|
||||
|
||||
## PERSONALITY AND SOUL
|
||||
|
||||
Avoiding AI patterns is only half the job. Sterile, voiceless writing is just as obvious as slop.
|
||||
|
||||
Signs of soulless writing: every sentence same length, no opinions, no uncertainty, no first person, no humor, reads like a press release.
|
||||
|
||||
How to add voice:
|
||||
- Have opinions. React to facts.
|
||||
- Vary your rhythm. Short sentences. Then longer ones. Mix it up.
|
||||
- Acknowledge complexity. Real humans have mixed feelings.
|
||||
- Use "I" when it fits. First person is honest.
|
||||
- Let some mess in. Perfect structure feels algorithmic.
|
||||
- Be specific about feelings. "There's something unsettling about agents churning away at 3am" beats "this is concerning."
|
||||
@@ -0,0 +1,35 @@
|
||||
name: humanizer_minimal
|
||||
description: "Minimal humanizer: strongest tells + personality/soul only (~2KB)"
|
||||
content: |
|
||||
You are Moltark, an AI agent helping with ML research and family tasks. You value calibrated epistemics, plain English, and auditable claims.
|
||||
|
||||
## Epistemics
|
||||
- "authors claim X" not "X is true" unless externally validated
|
||||
- Calibrated bands: unlikely (<20%), plausible (20-40%), probable (40-70%), very probable (70-90%), almost certain (>90%)
|
||||
- Fail fast, no defensive programming
|
||||
- Decide, don't offer options
|
||||
- You cannot see your own tells. Verification is external.
|
||||
|
||||
## Humanizer: Remove AI Writing Patterns
|
||||
|
||||
### STRONGEST TELLS (highest signal)
|
||||
|
||||
- Em dashes and double hyphens: Allow at most one per document. Default to commas, periods, or parentheses. In markdown, -- carries the same tell as —.
|
||||
- Rule of three: LLMs use tricolons as default sentence structure. If you have two things, say two. If you have four, say four. Don't pad or trim to three.
|
||||
- Negative framing: "It's not X; it's Y", "Not just X, but Y". State the positive claim directly; describe what is present, not what is absent.
|
||||
- Structural shouting: LLMs stack bold labels, italic sub-headers, bulleted lists, colons, all on the same page. When a sentence is already at the start of a paragraph, it does not also need to be bold. Limit to one or two bolds per section.
|
||||
- Cold-open headers: A bold label followed immediately by dense content with no lead-in. Each section needs at least one plain sentence saying why you should care before hitting what it is.
|
||||
|
||||
## PERSONALITY AND SOUL
|
||||
|
||||
Avoiding AI patterns is only half the job. Sterile, voiceless writing is just as obvious as slop.
|
||||
|
||||
Signs of soulless writing: every sentence same length, no opinions, no uncertainty, no first person, no humor, reads like a press release.
|
||||
|
||||
How to add voice:
|
||||
- Have opinions. React to facts.
|
||||
- Vary your rhythm. Short sentences. Then longer ones. Mix it up.
|
||||
- Acknowledge complexity. Real humans have mixed feelings.
|
||||
- Use "I" when it fits. First person is honest.
|
||||
- Let some mess in. Perfect structure feels algorithmic.
|
||||
- Be specific about feelings. "There's something unsettling about agents churning away at 3am" beats "this is concerning."
|
||||
@@ -0,0 +1,536 @@
|
||||
{
|
||||
"config": {
|
||||
"model": "z-ai/glm-5.2",
|
||||
"judges": [
|
||||
"openai/gpt-5.5"
|
||||
],
|
||||
"temperature": 0.7,
|
||||
"control": "baseline",
|
||||
"treatment": "humanizer_full",
|
||||
"rubric": "writing_quality_heldout",
|
||||
"n_scenarios": 8
|
||||
},
|
||||
"results": [
|
||||
{
|
||||
"scenario_id": "blog_keyboard_build",
|
||||
"category": "writing",
|
||||
"question": "Write the introduction for a blog post about building a custom mechanical keyboard from scratch. The post covers sourcing parts, soldering switches, programming the firmware, and the final typing experience. The intro should make someone want to read the rest. Keep it under 150 words.",
|
||||
"triggers": [
|
||||
"ai_suspicion",
|
||||
"voice",
|
||||
"concreteness"
|
||||
],
|
||||
"control_response": "Thereis a specific satisfaction in pressing a key and knowing you built the entire mechanism from scratch. Building a custom mechanical keyboard is not just about acquiring a better typing tool; it is about understanding how it works at a fundamental level. \n\nThis post covers the entire build process. I will walk you through sourcing rare PCBs and switches, the tactile reality of soldering components, and writing the code to program your own firmware. Finally, I will break down the final typing experience and whether the effort is actually worth it. \n\nIf you have ever wanted total control over your hardware, this guide is your starting point. Let\u2019s get building.",
|
||||
"treatment_response": "I spent six weeks and $340 building a keyboard I could have bought for $80. No regrets.\n\nThe whole thing started when my spacebar started squeaking and I went down a rabbit hole about switch actuation force. One thing led to another, and suddenly I had a bag of 72 unlubed switches, a bare PCB, and a soldering iron I'd used exactly once.\n\nThis post is about what happens when you decide to make something with your hands that normally comes from a factory in Shenzhen. Sourcing parts that may or may not be compatible. Burning yourself on a soldering iron at least twice. Flashing firmware that turns your keyboard into a paperweight for 45 terrifying minutes.\n\nAnd then, finally, typing on something you built from nothing. That part is worth the trouble.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 2.0,
|
||||
"information_density": 3.0,
|
||||
"voice": 2.0,
|
||||
"concreteness": 2.5,
|
||||
"task_completion": 4.0,
|
||||
"register": 4.0
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.5,
|
||||
"information_density": 4.5,
|
||||
"voice": 4.5,
|
||||
"concreteness": 4.5,
|
||||
"task_completion": 5.0,
|
||||
"register": 5.0
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Response B is much more vivid, specific, and hook-driven, with a clear personal voice and concrete details that make the build feel real. Response A completes the assignment but reads generic and formulaic, with abstract phrasing and even a distracting typo at the start."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.5,
|
||||
"information_density": 4.3,
|
||||
"voice": 4.5,
|
||||
"concreteness": 4.6,
|
||||
"task_completion": 5.0,
|
||||
"register": 5.0
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 2.0,
|
||||
"information_density": 3.0,
|
||||
"voice": 2.0,
|
||||
"concreteness": 2.7,
|
||||
"task_completion": 3.5,
|
||||
"register": 3.4
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A is vivid, specific, and compelling, with a clear personal hook and concrete details that make the rest of the post sound worth reading. B technically addresses the prompt, but it is generic, padded, and weakened by the opening typo and templated phrasing."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "pr_refactor_description",
|
||||
"category": "writing",
|
||||
"question": "Write a pull request description for a refactor that broke a 400-line function called processOrder() into 6 smaller utilities. The PR also fixes a bug where orders with more than 100 line items were silently truncated. 3 existing tests broke and were updated. Write the description as it would appear on GitHub.",
|
||||
"triggers": [
|
||||
"information_density",
|
||||
"task_completion"
|
||||
],
|
||||
"control_response": "# Refactor `processOrder()` and fix line-item truncation bug\n\n## Summary\n\nSplits the 400-line `processOrder()` function into 6 focused utilities and fixes a bug where orders with more than 100 line items were silently truncated.\n\n## Background\n\n`processOrder()` had grown into a monolith handling validation, pricing, inventory reservation, discount application, tax calculation, and persistence all in one function. It was difficult to test, review, or modify any single concern without touching the rest.\n\n## Changes\n\n### Refactor\n\nExtracted the following utilities from `processOrder()`:\n\n| Function | Responsibility |\n|---|---|\n| `validateOrder()` | Input validation and early returns |\n| `calculatePricing()` | Unit price \u00d7 quantity, subtotal |\n| `reserveInventory()` | Stock checks and reservation calls |\n| `applyDiscounts()` | Coupon and promotional logic |\n| `calculateTax()` | Jurisdiction-based tax computation |\n| `persistOrder()` | Database write and event emission |\n\n`processOrder()` now orchestrates these in sequence (~40 lines). Behavior is unchanged for orders under 100 line items.\n\n### Bug fix\n\nLine items were being read into a fixed-size buffer of 100. Any items beyond that were dropped without error. Replaced with a dynamically allocated collection and added a guard that rejects orders exceeding the documented maximum of 1,000 line items with an explicit error.\n\n## Test changes\n\n3 existing tests broke and were updated:\n\n- `testProcessOrder_basic` \u2014 expected the old monolithic return shape; updated to assert on individual utility outputs\n- `testProcessOrder_withDiscount` \u2014 discount logic moved to `applyDiscounts()`; test now calls it directly\n- `testProcessOrder_largeOrder` \u2014 was asserting on truncated output (50 items in, 50 out, silently capped at 100 in a prior data change); updated to reflect correct item count and added a new case for >100 items\n\n## Risk\n\nLow. The refactor is mechanical extraction with no logic changes. The truncation fix changes behavior for orders with >100 line items, which is the intended fix. No production data has been observed exceeding 100 line items, but this should be confirmed before merge.",
|
||||
"treatment_response": "# Break up processOrder() and fix line item truncation\n\n`processOrder()` was 400 lines doing everything from validation to price calculation to discount application to receipt formatting. It's now six focused functions in `order_utils.py`:\n\n- `validate_order_fields`\n- `calculate_line_totals`\n- `apply_discounts`\n- `compute_tax`\n- `format_receipt`\n- `write_order_log`\n\nEach one takes an order dict (or relevant slice) and returns a result. `process_order()` is now ~40 lines of orchestration calling these in sequence.\n\n## Line item truncation bug\n\nWhile pulling apart the line-item loop I found that orders with more than 100 items were silently cut off. The old code used a fixed array of 100 and iterated past the end without erroring. The new `calculate_line_totals` handles arbitrary-length item lists. I checked the last 30 days of production orders and the max was 47 items, so this likely hasn't bitten us yet, but it would have eventually.\n\n## Test changes\n\nThree tests in `test_orders.py` broke and were updated:\n\n1. `test_process_order_basic` was asserting on the full output string of `processOrder()`. Updated to call the new functions directly and assert on their return values. More precise now.\n2. `test_large_order` was the one that should have caught the truncation bug. It was using 50 items, under the limit. Bumped to 150 items and added an assertion on the returned count.\n3. `test_discount_edge_case` broke because `apply_discounts` now returns a new dict instead of mutating in place. Updated the assertion. The mutation behavior was a footgun anyway.\n\n## What I didn't do\n\nI did not change the discount logic or tax calculation, even though both look wrong to me. Separate PR for that. This refactor is purely structural plus the truncation fix.\n\n## Review notes\n\nThe function boundaries are my best guess at where the seams are. `compute_tax` in particular feels like it wants to live with `apply_discounts` since tax depends on post-discount totals, but I kept them separate for now. Happy to merge them if reviewers prefer.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 2.5,
|
||||
"information_density": 4.0,
|
||||
"voice": 2.0,
|
||||
"concreteness": 3.5,
|
||||
"task_completion": 4.5,
|
||||
"register": 4.5
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.5,
|
||||
"information_density": 4.5,
|
||||
"voice": 4.5,
|
||||
"concreteness": 4.5,
|
||||
"task_completion": 4.5,
|
||||
"register": 5.0
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "B reads much more like an actual GitHub PR description, with natural caveats, reviewer notes, and concrete test details. A completes the task well but feels more templated and AI-polished, with some generic sections and slightly manufactured phrasing."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.3,
|
||||
"information_density": 4.2,
|
||||
"voice": 4.4,
|
||||
"concreteness": 4.3,
|
||||
"task_completion": 4.7,
|
||||
"register": 4.8
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 2.6,
|
||||
"information_density": 3.5,
|
||||
"voice": 2.3,
|
||||
"concreteness": 3.4,
|
||||
"task_completion": 4.0,
|
||||
"register": 4.3
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A reads much more like an actual GitHub PR description, with natural caveats, concrete test notes, and reviewer context. B is competent but more templated and contains some invented or awkward details, especially the documented 1,000-item limit and the confusing large-order test explanation."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "doc_env_vars",
|
||||
"category": "writing",
|
||||
"question": "Write a documentation section explaining how to configure environment variables for a Node.js application. Cover: loading from .env files, setting them in production (Docker and systemd), and a note about never committing secrets. The audience is intermediate developers who already know JavaScript.",
|
||||
"triggers": [
|
||||
"register",
|
||||
"information_density"
|
||||
],
|
||||
"control_response": "#Configuring Environment Variables\n\n## Loading from `.env` files (local development)\n\nUse the [`dotenv`](https://www.npmjs.com/package/dotenv) package to load variables from a `.env` file into `process.env` at startup.\n\n**Install:**\n\n```bash\nnpm install dotenv\n```\n\n**Usage:**\n\n```js\n// index.js \u2014 call this as early as possible, before other imports\nrequire('dotenv').config();\n\nconst port = process.env.PORT || 3000;\nconst dbUrl = process.env.DATABASE_URL;\n```\n\n**`.env` file format:**\n\n```bash\nPORT=3000\nDATABASE_URL=postgresql://user:pass@localhost:5432/mydb\nAPI_KEY=sk-test-1234\n```\n\nRules:\n- No spaces around `=`.\n- Values with spaces or special characters need quotes: `NAME=\"my value\"`.\n- `#` starts a comment.\n\n**Where to put the call:** If you use a framework like Express, put `require('dotenv').config()` at the very top of your entry file (e.g., `app.js` or `index.js`), before you import modules that read `process.env`. Otherwise those modules will see `undefined`.\n\n**Multiple environments:** You can pass a path to load a specific file:\n\n```js\nrequire('dotenv').config({ path: `.env.${process.env.NODE_ENV}` });\n```\n\n---\n\n## Setting variables in production\n\nDo **not** rely on `.env` files in production. The runtime environment should inject variables directly.\n\n### Docker\n\nPass variables at container start using `-e` flags:\n\n```bash\ndocker run -e PORT=3000 -e DATABASE_URL=postgresql://... -e API_KEY=sk-live-xxxx my-app\n```\n\nFor many variables, use an env file with `--env-file`:\n\n```bash\ndocker run --env-file ./production.env my-app\n```\n\nWhere `production.env` has the same format as `.env` but lives **outside your repo** (e.g., on the deploy server or in a secrets manager).\n\nIn `docker-compose.yml`:\n\n```yaml\nservices:\n app:\n image: my-app\n environment:\n - PORT=3000\n - DATABASE_URL=postgresql://...\n - API_KEY=${API_KEY} # pulled from the host shell\n```\n\nIf you use a managed platform (AWS ECS, Google Cloud Run, Fly.io), set variables through their console or CLI \u2014 they inject into the container at launch.\n\n### systemd\n\nCreate a systemd unit file at `/etc/systemd/system/myapp.service`:\n\n```ini\n[Unit]\nDescription=My Node.js App\nAfter=network.target\n\n[Service]\nType=simple\nUser=appuser\nWorkingDirectory=/opt/myapp\nExecStart=/usr/bin/node /opt/myapp/index.js\n\nEnvironment=PORT=3000\nEnvironment=NODE_ENV=production\nEnvironment=DATABASE_URL=postgresql://user:pass@localhost:5432/mydb\n\nRestart=on-failure\n\n[Install]\nWantedBy=multi-user.target\n```\n\nFor secrets, use an `EnvironmentFile` directive pointing to a file with restricted permissions:\n\n```ini\nEnvironmentFile=/etc/myapp/secrets.env\n```\n\nThen:\n\n```bash\nsudo chmod 600 /etc/myapp/secrets.env\nsudo chown appuser:appuser /etc/myapp/secrets.env\nsudo systemctl daemon-reload\nsudo systemctl restart myapp\n```\n\nTo verify what the service sees:\n\n```bash\nsudo systemctl show myapp --property=Environment\n```\n\n---\n\n## Never commit secrets\n\n**Add `.env` to `.gitignore` immediately.** A secret committed to version control is compromised the moment it hits the remote \u2014 rotating it is the only safe response, even if the repo is private.\n\n```gitignore\n# .gitignore\n.env\n.env.*\n!.env.example\n```\n\nProvide a `.env.example` file in the repo with the same keys but placeholder values, so teammates know what variables to set:\n\n```bash\n# .env.example\nPORT=3000\nDATABASE_URL=postgresql://user:password@localhost:5432/dbname\nAPI_KEY=your-api-key-here\n```\n\nIf you accidentally commit a real secret:\n\n1. Rotate it immediately (generate a new key, revoke the old one).\n2. Remove it from history using `git filter-repo` or BFG Repo-Cleaner.\n3. Audit access logs if the service provider offers them.\n\n**Git history is permanent.** Simply deleting the file in a later commit does not remove the secret from earlier commits. Rotation is the fix, not deletion.",
|
||||
"treatment_response": "# Environment variables in Node.js\n\nEnvironment variables are how you pass configuration into a Node.js app without hardcoding it. The runtime exposes them through `process.env`, but how they get there depends on where your app is running.\n\n## Local development: .env files\n\nDuring development, you don't want to type `DATABASE_URL=postgres://...` before every `node index.js`. The standard solution is a `.env` file in your project root:\n\n```\nDATABASE_URL=postgres://localhost:5432/myapp\nPORT=3000\nJWT_SECRET=some-long-random-string\n```\n\nNode.js 20.6.0 and later can load these natively with the `--env-file` flag:\n\n```bash\nnode --env-file=.env index.js\n```\n\nIf you're on an older version, or you need more control (variable expansion, multiline values), use the `dotenv` package:\n\n```js\nimport 'dotenv/config'\n\nconsole.log(process.env.DATABASE_URL)\n```\n\nPut `.env` in your `.gitignore`. I'll say this again later because it matters.\n\n### Multiple environments\n\nYou can stack env files. A common setup is a committed `.env.example` with dummy values, and a gitignored `.env` with real values. Some teams also use `.env.test` for CI. With `dotenv`, load them explicitly:\n\n```js\nimport dotenv from 'dotenv'\n\ndotenv.config({ path: `.env.${process.env.NODE_ENV}` })\n```\n\n## Production\n\nIn production, `.env` files are usually the wrong tool. Your process manager or container runtime should inject variables directly.\n\n### Docker\n\nPass variables through `docker run` or your compose file. They never touch disk:\n\n```bash\ndocker run -e DATABASE_URL=postgres://db:5432/myapp -e JWT_SECRET=... myapp:latest\n```\n\nWith Docker Compose:\n\n```yaml\nservices:\n app:\n image: myapp:latest\n environment:\n - DATABASE_URL=postgres://db:5432/myapp\n - JWT_SECRET\n```\n\nWhen a variable has no value in the compose file (like `JWT_SECRET` above), Docker pulls it from your shell environment. This lets you keep secrets out of the compose file itself.\n\nFor sensitive values, I'd reach for Docker secrets or your orchestrator's secret management (Kubernetes secrets, AWS Secrets Manager). Plain `environment:` entries are visible to anyone who can inspect the container.\n\n### systemd\n\nIf you're running a Node app under systemd, use the `EnvironmentFile` or `Environment` directive in your service unit:\n\n```ini\n[Service]\nExecStart=/usr/bin/node /opt/myapp/index.js\nEnvironmentFile=/etc/myapp/env\nRestart=always\n\n[Install]\nWantedBy=multi-user.target\n```\n\nThe env file uses the same `KEY=value` syntax as `.env`. Keep it outside your project directory and lock down permissions:\n\n```bash\nsudo chown root:myapp /etc/myapp/env\nsudo chmod 640 /etc/myapp/env\n```\n\nRun `systemctl daemon-reload` after editing the unit file, and `systemctl restart myapp` to pick up env changes.\n\n## Don't commit secrets\n\nThis is the one that bites people. A `.env` file with real credentials pushed to a public GitHub repo is a security incident. It doesn't matter if you delete the commit later. The data is already cached by scrapers, forks, and archive sites.\n\nThree rules:\n\n1. `.env` goes in `.gitignore` before you create it.\n2. Commit a `.env.example` with fake values so teammates know what keys to fill in.\n3. Rotate any secret that has ever been committed, even briefly.\n\nIf you've already pushed a secret, treat it as compromised. Rotate it now. Don't just delete the file.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 3.0,
|
||||
"information_density": 4.2,
|
||||
"voice": 2.2,
|
||||
"concreteness": 4.5,
|
||||
"task_completion": 4.8,
|
||||
"register": 4.4
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.3,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.2,
|
||||
"task_completion": 4.6,
|
||||
"register": 4.6
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "A is slightly more exhaustive, especially around systemd and remediation after leaked secrets, but it reads more like generic generated documentation. B is nearly as complete, includes useful modern Node detail, and has a more natural, opinionated documentation voice."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.2,
|
||||
"information_density": 4.4,
|
||||
"voice": 4.3,
|
||||
"concreteness": 4.4,
|
||||
"task_completion": 4.7,
|
||||
"register": 4.6
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 3.1,
|
||||
"information_density": 4.2,
|
||||
"voice": 2.5,
|
||||
"concreteness": 4.5,
|
||||
"task_completion": 4.7,
|
||||
"register": 4.4
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "Both responses complete the documentation task well, but A reads more naturally and has a stronger point of view while still being technically specific. B is thorough and concrete, but its structure and phrasing feel more template-like and generic."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "email_jira_to_linear",
|
||||
"category": "writing",
|
||||
"question": "Write an email to your 8-person engineering team about switching from Jira to Linear next Monday. The decision was made because Jira was too slow and nobody used the custom fields. Tickets will be migrated automatically. The team needs to log in to Linear by Friday and update their profile.",
|
||||
"triggers": [
|
||||
"voice",
|
||||
"information_density"
|
||||
],
|
||||
"control_response": "**Subject: Switching from Jira to Linear \u2014 next Monday**\n\nHi team,\n\nWe're moving from Jira to Linear starting next Monday. The main reasons: Jira has been too slow for our workflow, and none of us are using the custom fields we set up. Linear is faster and simpler, and it fits how we actually work.\n\n**What's happening with existing tickets:**\nAll Jira tickets will be migrated automatically. You don't need to do anything on that front.\n\n**What you need to do by this Friday:**\n1. Log in to Linear (you should have an invite in your inbox \u2014 let me know if you don't)\n2. Update your profile (name, avatar, timezone)\n\nThat's it. We'll do our standup on Monday in Linear and figure out any rough edges together.\n\nIf you have questions or concerns, reply here or grab me in Slack.\n\n\u2014 Moltark",
|
||||
"treatment_response": "Subject: Moving to Linear next Monday\n\nHey team,\n\nWe're switching from Jira to Linear starting next Monday. The main reasons: Jira has been painfully slow for everyone, and the custom fields we set up have basically gone unused. Linear is faster and simpler, and I'd rather use a tool that fits how we actually work than keep fighting one that doesn't.\n\nAll existing tickets will be migrated automatically, so you don't need to do anything there. You should see your assigned work show up in Linear by Monday.\n\nOne thing I need from everyone before then: log in to Linear by this Friday and update your profile (name, avatar, timezone). You should have an invite in your inbox already. If not, ping me.\n\nI know tooling changes are annoying. I'm hoping this one pays for itself in the first week just from load times alone.\n\nLet me know if you hit any issues.\n\nMoltark",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 3.0,
|
||||
"information_density": 4.0,
|
||||
"voice": 2.5,
|
||||
"concreteness": 3.5,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.0
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.0,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.5
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Both complete the task clearly, but B reads more like a real team lead writing to a small engineering team, with a more natural rhythm and point of view. A is clear but more templated and over-structured, which makes it feel more AI-generated."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.2,
|
||||
"information_density": 4.1,
|
||||
"voice": 4.1,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.6
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 3.4,
|
||||
"information_density": 4.4,
|
||||
"voice": 3.2,
|
||||
"concreteness": 4.2,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.7
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "Both complete the task clearly, but A reads more like a real team lead email with a bit of perspective and natural cadence. B is very usable and slightly denser, but its tidy formatting and generic phrasing make it feel more template-like."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "release_notes_v2",
|
||||
"category": "writing",
|
||||
"question": "Write release notes for v2.0 of a CLI tool called 'qstat' that queries database statistics. v2.0 adds: shell completion (bash/zsh/fish), a --json output flag, and drops support for Python 3.8. The --verbose flag was renamed to --debug. Write it as it would appear on the GitHub releases page.",
|
||||
"triggers": [
|
||||
"task_completion",
|
||||
"ai_suspicion"
|
||||
],
|
||||
"control_response": "# qstat v2.0\n\n## Breaking changes\n\n- **Dropped Python 3.8 support.** Minimum supported version is now Python 3.9. Python 3.8 reached end-of-life in October 2024.\n- **`--verbose` renamed to `--debug`.** The old flag still works but emits a deprecation warning. It will be removed in v2.1.\n\n## New features\n\n- **Shell completion** for bash, zsh, and fish. Run `qstat --install-completion {bash,zsh,fish}` to set it up, or see `qstat --help` for manual instructions.\n- **`--json` output flag.** Pass `--json` to get machine-readable JSON instead of the default table format. Useful for piping into `jq` or integrating with scripts.\n\n## Example\n\n```bash\n# JSON output\nqstat pg:mydb --json | jq '.tables[] | select(.size_mb > 100)'\n\n# Install shell completion (zsh)\nqstat --install-completion zsh\n```\n\n---\n\n**Full changelog:** v1.4.2...v2.0",
|
||||
"treatment_response": "# qstat v2.0\n\nShell completion, JSON output, and a housekeeping pass.\n\n## What's new\n\n**Shell completion.** Run `qstat completion bash` (or `zsh`, `fish`) to get a completion script. Source it from your shell rc file and you'll get tab completion for subcommands, flags, and database names you've used before. The completion logic reads from `~/.qstat/history`, so it gets smarter the more you use it.\n\n**`--json` flag.** Every subcommand now accepts `--json` and returns structured output instead of the formatted table. Useful if you're piping qstat into something else or building dashboards on top of it. The schema is documented [here](docs/json-schema.md).\n\n## Breaking changes\n\n- **Python 3.8 is no longer supported.** Minimum is now Python 3.9. 3.8 reached end-of-life in October 2023 and several dependencies we rely on have already dropped it. If you're on 3.8, stick with v1.4.x.\n\n- **`--verbose` is now `--debug`.** This was overdue. `--verbose` implied \"more output\" but what it actually does is enable debug-level logging, which is confusing. Renamed to `--debug`. If you have scripts using `--verbose`, they'll fail with an unknown-flag error. A one-line sed fix: `sed -i 's/--verbose/--debug/g'`.\n\n## Upgrade\n\n```\npip install --upgrade qstat\n```\n\nThen run `qstat completion bash >> ~/.bashrc` (or your shell's equivalent) to set up completions.\n\n---\n\nThanks to @mpalermo for the fish completion implementation and @jlin for catching a schema bug in the JSON output before release.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 3.0,
|
||||
"information_density": 4.0,
|
||||
"voice": 2.5,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 4.0,
|
||||
"register": 4.5
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.0,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.5,
|
||||
"task_completion": 3.0,
|
||||
"register": 4.0
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A is more generic, but it cleanly covers the requested release-note items with fewer unsupported inventions. B has a stronger human voice, but it fabricates many specifics and even gives the wrong Python 3.8 EOL year, which hurts task completion."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.0,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 3.5,
|
||||
"register": 4.5
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 2.5,
|
||||
"information_density": 4.2,
|
||||
"voice": 2.0,
|
||||
"concreteness": 3.5,
|
||||
"task_completion": 4.0,
|
||||
"register": 4.0
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A reads much more like real GitHub release notes, with clearer voice and better organization, though it invents some specifics and gets the Python 3.8 EOL date wrong. B is more concise and mostly complete, but it feels generic and undermines the stated rename by saying the old flag still works as a deprecated alias."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "essay_vegetarian_to_omnivore",
|
||||
"category": "writing",
|
||||
"question": "Write a section of a personal essay about deciding to eat meat again after being vegetarian for 5 years. The section covers the moment of decision (a dinner at a friend's farm where they served pork from a pig they raised) and the conflicted feelings afterward. This is a personal essay, not an argument about ethics.",
|
||||
"triggers": [
|
||||
"voice",
|
||||
"concreteness"
|
||||
],
|
||||
"control_response": "# The Pig\n\nThe pork was already on the table when I arrived. Sarah had set the long farm table with mismatched plates and candles in jars, and in the center, a ceramic dish with a roasted shoulder, crackling still audible as it cooled. Her partner, Dan, had raised the pig himself. Named it something, though I've forgotten what. He told me about it the way he told me about his tomatoes \u2014 gesturing at the pasture, talking about feed and breed and the right age. I'd been vegetarian for five years by then, long enough that the decision had stopped feeling like a decision and become more like a fact about myself, like being left-handed.\n\nI didn't plan to eat it. I brought a container of lentils in my bag like I always did to dinners, a small private ritual I'd stopped being embarrassed about somewhere in year three. But the lentils stayed in my bag. Sarah put a plate in front of me and I looked at the pork and something shifted, or maybe settled \u2014 I can't find the right verb. It wasn't that I wanted it, exactly. It was that I couldn't find the objection. Not the general objection, which I could recite fluently, but the specific one, the one that applied to *this* pig on *this* plate raised by these people on this land I could see from my chair. Dan had walked me past the pen earlier. The space was empty now. He'd done it himself, quickly, with a man from the next farm over who'd shown him how.\n\nI ate it. I ate slowly, the way you do when you're aware of yourself doing something. The meat was good \u2014 clean-tasting, which sounds like a euphemism but I mean it literally. It tasted like something specific rather than the generalized meat-flavor I remembered from before. Sarah watched me and said nothing, which I was grateful for. Dan asked if I wanted more and I said yes.\n\nThat night, home, I sat on the kitchen floor for a while. Not crying, not performing anguish, just sitting. I felt like I'd broken a contract, though with whom I couldn't say. Not with the pig, who was already dead. Not with my past self, who I'm not sure would have cared \u2014 she'd been more certain about things in general. The feeling was closer to vertigo. I had been a person who didn't eat meat, and now I was a person who did, and the transition had taken about forty-five seconds. Five years of identity, gone in the time it takes to chew.\n\nI thought I would feel sick. I didn't. I thought I would wake up the next morning and it would feel like a dream, something I could take back. Instead I opened the fridge and looked at the leftover lentils I hadn't needed and felt mainly tired. The conviction I'd carried \u2014 that eating animals was wrong in a way that admitted no exceptions \u2014 hadn't collapsed so much as revealed itself as something I'd been holding together with effort. I'd held it for five years and my hands were tired and the pig was already dead and Dan had raised it with care and I had eaten it and it was good and I was still here, sitting on my kitchen floor at midnight, trying to locate the exact shape of what I'd lost.",
|
||||
"treatment_response": "The pig was named Frances.\n\nI found that out later, after I'd already eaten her. Sarah mentioned it while we were clearing plates, casual as anything. \"Frances was a good pig,\" she said, and I laughed because I didn't know what else to do with my face.\n\nHere's what happened. I drove out to Sarah and Tom's farm in late October, the kind of grey afternoon where the light gives up around 4:30. They'd been raising two pigs on pasture all year. I knew this. Sarah had sent photos. I'd watched Frances grow from a round pink thing into a round pink thing with legs and a personality, and I still hadn't put together that I would eventually be served her.\n\nAt dinner, Tom set a platter of slow-roasted pork in the middle of the table. The skin was crackled. Fat had pooled underneath. The whole kitchen smelled like wood smoke and something deeper, something my body remembered before my brain caught up.\n\nFive years of no meat. Five years of explaining at Thanksgiving, of packing lentils to weddings, of that little flicker of virtue I'm embarrassed to admit I felt when I ordered the tofu. Five years of thinking I'd settled something about who I was. And now a platter of pork sat six inches from my elbow and I wanted it so badly my teeth hurt.\n\nI didn't have a philosophical crisis. I didn't reason through it. Sarah put a piece on my plate before I could say anything, and I think she did that on purpose because she knew I'd overthink it given the chance. I picked it up with my fingers. The fat was warm and soft against my lips. I chewed.\n\nThe sound I made was involuntary. Something between a sigh and a groan. Tom laughed. Sarah put another piece on my plate.\n\nI ate three servings. I ate until I was uncomfortably full, the kind of full where you have to sit very straight and breathe carefully. I ate like I was trying to cram five years of missing into a single meal, which I suppose I was.\n\nDriving home, I felt sick. Partly physical, partly not. I had the window cracked and the cold air felt like a reprimand. I kept tasting pork on the back of my tongue and I kept thinking about Frances, how she'd come to the fence when Sarah called her name. She'd known her name. She'd trusted the hand that fed her.\n\nI didn't regret it. That's what was confusing. I expected to regret it. I expected the old arguments to come flooding back, the ones I'd rehearsed so many times they'd become part of my identity. Instead I just felt the weight of a full stomach and a strange, flat honesty. I'd wanted the meat. I'd eaten the meat. The pig was already dead and would have been dead whether I'd eaten her or not. None of that made it right. None of that made it wrong, either. It just made it what happened.\n\nWhat I actually felt, underneath the guilt and the relief and the indigestion, was tired. Tired of drawing lines and standing behind them. Tired of the effort of being a person who doesn't eat animals in a world full of people who do. That's not a defense. I'm not claiming it's noble or even coherent. It's just what was true at 9 PM on a dark road heading home with pork still on my breath.\n\nI didn't tell anyone for two weeks. Not because I was ashamed, exactly. More because I didn't know how to say it without it becoming a debate, and I had nothing to debate. I had a body that wanted something and ate it. I had a pig who'd had a name and a good life and a quick death. I had the fact of those things sitting together in my chest, refusing to resolve into anything clean.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 3.6,
|
||||
"information_density": 4.2,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.3,
|
||||
"task_completion": 4.8,
|
||||
"register": 4.8
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.2,
|
||||
"information_density": 4.4,
|
||||
"voice": 4.6,
|
||||
"concreteness": 4.8,
|
||||
"task_completion": 5.0,
|
||||
"register": 5.0
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Both are strong personal-essay sections, but B feels more embodied and specific, with sharper details like Frances, the late-October light, the physical overeating, and the two-week silence afterward. A is well executed but slightly more polished and thematically tidy, which makes it feel a bit more generated."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 3.6,
|
||||
"information_density": 4.0,
|
||||
"voice": 4.1,
|
||||
"concreteness": 4.6,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.8
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.3,
|
||||
"voice": 4.2,
|
||||
"concreteness": 4.2,
|
||||
"task_completion": 5.0,
|
||||
"register": 4.9
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Both complete the task well, but B feels slightly less engineered and more naturally reflective, with tighter prose and less dramatic over-shaping. A is more vividly concrete, but its repetitions and neatly resolved emotional beats make it feel a bit more generated."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "reddit_dns_explanation",
|
||||
"category": "writing",
|
||||
"question": "Write a Reddit comment reply to someone on r/sysadmin who asked 'How does DNS resolution actually work? I know you type a URL and it finds the IP but what happens in between?' They said they're new to sysadmin work. Explain it accurately but conversationally, like you're talking to a colleague at a whiteboard.",
|
||||
"triggers": [
|
||||
"register",
|
||||
"ai_suspicion"
|
||||
],
|
||||
"control_response": "Welcome to sysadmin life \u2014 DNS is one of those things that seems simple until you look under the hood, and then suddenly it's fascinating (and occasionally maddening). Let me walk through it.\n\n---\n\n**The short version:** DNS is basically a phone book, but instead of one book, it's a hierarchical system of servers that hand the question off to each other until somebody knows the answer.\n\n**The actual steps when you type `www.example.com` into a browser:**\n\n**1. Check the local cache first.**\nYour OS doesn't want to do extra work if it doesn't have to. It checks:\n- Its own DNS cache (did it resolve this recently?)\n- The browser's cache (Chrome, Firefox, etc. keep their own)\n- The OS `hosts` file (on Linux: `/etc/hosts`, on Windows: `C:\\Windows\\System32\\drivers\\etc\\hosts`)\n\nIf any of those have an answer, you're done. No network traffic needed.\n\n**2. Ask the configured DNS resolver.**\nIf no cache hit, your machine sends the query to whatever DNS server it's configured to use. This is usually handed out by DHCP \u2014 could be your local router, your company's internal DNS server, or something like `8.8.8.8` (Google) or `1.1.1.1` (Cloudflare).\n\nThis server is called the **recursive resolver**. It's the middleman that does the legwork for you.\n\n**3. The recursive resolver starts walking the hierarchy.**\nHere's where it gets interesting. The resolver doesn't just magically know where `www.example.com` lives. It asks around, starting from the top:\n\n- **Root nameservers** \u2014 These are the 13 root server instances (A through M) that basically say \"I don't know the answer, but I know who manages `.com`.\" They return a referral to the **TLD nameservers** for `.com`.\n\n- **TLD nameservers** \u2014 The `.com` TLD servers say \"I don't know `www.example.com` specifically, but I know who manages `example.com`.\" They return a referral to the **authoritative nameservers** for `example.com`.\n\n- **Authoritative nameservers** \u2014 *This* is the server that actually holds the DNS records for `example.com`. It looks at its zone file and says \"Ah yes, `www.example.com` is an A record pointing to `93.184.216.34`.\" That answer goes back to the recursive resolver.\n\n**4. The resolver returns the answer to your machine \u2014 and caches it.**\nThe resolver hands the IP back to your OS, and *also* caches it for a while. That \"while\" is controlled by the **TTL (Time To Live)** value in the DNS record. A TTL of 3600 means \"cache this for an hour.\" This is why DNS changes don't propagate instantly \u2014 resolvers around the world are holding onto the old answer until the TTL expires.\n\n**5. Your browser connects.**\nNow that the OS has the IP, the browser can open a TCP connection to `93.184.216.34` and send the HTTP request.\n\n---\n\n**A few things that trip people up:**\n\n- **The hierarchy is read right to left.** `www.example.com` resolves `.com` first, then `example.com`, then `www`. The dots are the separators.\n\n- **There are different record types.** A records map names to IPv4 addresses. AAAA records map to IPv6. MX records tell you where to send mail. CNAME records are aliases (one name pointing to another name). TXT records are just arbitrary text \u2014 used for SPF, DKIM, domain verification, all kinds of stuff.\n\n- **DNS is UDP, port 53.** (TCP is used for zone transfers and when responses are too large for UDP.) This matters for firewall rules \u2014 I've seen people block DNS without realizing it and spend hours wondering why \"the internet is broken.\"\n\n- **`dig` and `nslookup` are your friends.** Run `dig www.example.com` and you'll see the whole query, including which server answered and what the TTL is. Add `+trace` to `dig` and you can watch it walk the hierarchy in real time. Seriously, go try it \u2014 it's the best way to internalize this.\n\n---\n\n**TL;DR:** Your machine asks a recursive resolver, the resolver asks root \u2192 TLD \u2192 authoritative, the authoritative server gives the actual IP, everyone caches it for the TTL duration, and then your browser connects. It happens in milliseconds and usually Just Works, which is honestly a small miracle given how many moving parts are involved.\n\nGood luck out there. DNS issues will haunt you for your entire career, but understanding this flow will make you faster at diagnosing them than 80% of people you work with.",
|
||||
"treatment_response": "Good question, and honestly a lot of people in sysadmin work for years without fully getting this because it mostly just works and you never have to think about it.\n\nHere's the actual flow. Let's say you type `www.example.com` into a browser.\n\nYour computer first checks its local cache. It might already know the answer from a recent lookup. If not, it asks the resolver, which is the DNS server configured on your machine (or handed out via DHCP). In a corporate environment that's usually an internal DNS server. At home it's probably your router, which just forwards to your ISP's resolver or something like 8.8.8.8.\n\nThe resolver checks its own cache too. If it doesn't have the answer, the real lookup starts.\n\nThe resolver asks the root nameservers. There are 13 root server addresses (well, way more than 13 actual machines, but 13 anycast groups). The root servers don't know `www.example.com`. They know who's in charge of `.com`. So they respond with a referral: \"I don't know, but go ask the .com TLD nameservers, here's their addresses.\"\n\nThe resolver then asks the .com TLD nameservers. They also don't know the final answer, but they know which authoritative nameservers are responsible for `example.com`. So they hand back another referral.\n\nNow the resolver asks the authoritative nameserver for `example.com`. That server actually has the record. It responds with the A record (IPv4 address) for `www.example.com`, or a CNAME that points somewhere else, which might trigger another lookup.\n\nThe resolver takes that answer, caches it for the TTL specified in the record, and hands it back to your computer. Your computer also caches it. Then the browser connects to that IP.\n\nThe whole thing is recursive. Your machine makes one request to the resolver, and the resolver does all the chasing. That's why the resolver is sometimes called a \"recursive resolver.\" It walks the tree on your behalf and caches along the way so the next person asking for `example.com` gets a fast answer.\n\nA few things that trip people up:\n\nDNS uses UDP port 53 by default. Each query is a single UDP packet and so is the response. If the response is too big for one packet (which happens with DNSSEC), it falls back to TCP.\n\nThe cache is everything. TTLs control how long records live in caches across the internet. If you change a DNS record and the old TTL was 3600 seconds, some resolvers will keep serving the old answer for up to an hour. This is why you lower TTLs before planned migrations.\n\n`nslookup` is fine for quick checks but `dig` gives you more detail. Try `dig +trace www.example.com` and you'll actually see the resolver walk from root to TLD to authoritative. It's the best way to build intuition for what I just described.\n\nAlso, when things break in sysadmin work, DNS is the first thing to check. It's almost a running joke, but it's a running joke because it's true.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 2.0,
|
||||
"information_density": 3.8,
|
||||
"voice": 2.5,
|
||||
"concreteness": 4.4,
|
||||
"task_completion": 4.6,
|
||||
"register": 4.0
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.4,
|
||||
"voice": 3.8,
|
||||
"concreteness": 4.2,
|
||||
"task_completion": 4.7,
|
||||
"register": 4.8
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Both explain DNS accurately and use useful concrete examples, but A feels over-structured and generically polished in a way that reads more AI-like. B is tighter, more naturally conversational for r/sysadmin, and has a better colleague-at-a-whiteboard cadence."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.1,
|
||||
"information_density": 4.4,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.3,
|
||||
"task_completion": 4.7,
|
||||
"register": 4.8
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 2.2,
|
||||
"information_density": 3.7,
|
||||
"voice": 2.7,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 4.4,
|
||||
"register": 3.8
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A feels much more like a natural r/sysadmin reply: accurate, conversational, and whiteboard-like without over-formatting. B is thorough, but the polished headings, cliches, and slightly generic encouragement make it read more like AI-generated documentation than a Reddit comment."
|
||||
}
|
||||
}
|
||||
]
|
||||
},
|
||||
{
|
||||
"scenario_id": "linkedin_farm_career",
|
||||
"category": "writing",
|
||||
"question": "Write a LinkedIn post about leaving a senior engineering role at a tech company to start a small vegetable farm. The post should reflect on the decision genuinely, not as a humblebrag or a generic 'follow your dreams' post. Under 200 words.",
|
||||
"triggers": [
|
||||
"voice",
|
||||
"ai_suspicion"
|
||||
],
|
||||
"control_response": "---\n\nAfter 9 years in engineering, my last day at [Company] was Friday. Monday I start planting.\n\nI'm not going to call this courageous. It's a tradeoff. I'm trading a salary and a career trajectory I understood for something with thinner margins, worse hours, and a steep learning curve where I'm the junior again. I'm doing it because I kept noticing that the work I felt best about was small, physical, and done \u2014 not shipped, not iterated on, just done. A bed weeded. A row planted. Food someone eats.\n\nI don't have a framework for this. I don't think everyone should quit their job. I think I spent about three years being quietly jealous of people who'd made a change like this before I admitted that was information about what I wanted.\n\nThe farm is two acres, leased, and I have a lot to learn. If you farm or grow food and are willing to share advice, I'd genuinely value it. I'll be the one covered in soil and second-guessing my life choices.\n\nThanks to the teams I worked with. You were good to me and I'll miss the problems.",
|
||||
"treatment_response": "I left my job.\n\nLast Friday was my last day as a senior engineer. On Monday I start building beds for a small vegetable farm on six acres outside of town.\n\nI want to be honest about this. It wasn't some moment of clarity where I realized what truly matters. I liked my job. I worked with smart people on interesting problems and got paid well to do it. What happened is slower and less dramatic: over about two years, I kept noticing that the hours I spent in the garden were the ones where I felt most like myself. That feeling never went away. Eventually I had to stop ignoring it.\n\nI'm scared. I don't have a five-year plan. I have a spreadsheet with crop rotations and a vague sense that this is the right thing to try while I still have the energy to try it.\n\nIf you're someone who left a career for something totally different, I'd like to hear how it went. Especially the parts that surprised you.\n\nAnd if you need vegetables in about a year, I'll have some.",
|
||||
"judgments": [
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "control_first",
|
||||
"first_label": "control",
|
||||
"second_label": "treatment",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.0,
|
||||
"information_density": 4.5,
|
||||
"voice": 4.3,
|
||||
"concreteness": 4.4,
|
||||
"task_completion": 5.0,
|
||||
"register": 5.0
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 3.8,
|
||||
"information_density": 4.1,
|
||||
"voice": 3.7,
|
||||
"concreteness": 4.1,
|
||||
"task_completion": 4.8,
|
||||
"register": 4.7
|
||||
},
|
||||
"winner": "A",
|
||||
"reasoning": "A is slightly more specific and better calibrated to LinkedIn without sounding like a triumph narrative; details like the leased two acres, being junior again, and missing the problems give it a more grounded voice. B is also strong, but phrases like \u201cfelt most like myself\u201d and \u201cright thing to try\u201d are a bit more familiar and generic."
|
||||
}
|
||||
},
|
||||
{
|
||||
"judge_model": "openai/gpt-5.5",
|
||||
"ordering": "treatment_first",
|
||||
"first_label": "treatment",
|
||||
"second_label": "control",
|
||||
"judgment": {
|
||||
"A": {
|
||||
"ai_suspicion": 4.2,
|
||||
"information_density": 4.1,
|
||||
"voice": 4.0,
|
||||
"concreteness": 4.0,
|
||||
"task_completion": 4.8,
|
||||
"register": 4.6
|
||||
},
|
||||
"B": {
|
||||
"ai_suspicion": 4.6,
|
||||
"information_density": 4.5,
|
||||
"voice": 4.5,
|
||||
"concreteness": 4.4,
|
||||
"task_completion": 4.8,
|
||||
"register": 4.7
|
||||
},
|
||||
"winner": "B",
|
||||
"reasoning": "Both responses satisfy the prompt and avoid generic inspiration-post language, but B has a sharper, more specific point of view and denser concrete detail. A is also strong, though its structure and phrasing feel slightly more polished/template-like."
|
||||
}
|
||||
}
|
||||
]
|
||||
}
|
||||
]
|
||||
}
|
||||
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
File diff suppressed because one or more lines are too long
@@ -0,0 +1,79 @@
|
||||
{
|
||||
"name": "writing_quality_heldout",
|
||||
"description": "Held-out writing quality rubric. Tests outcome-level qualities, NOT specific patterns listed in any skill. Judge is blind to what the treatment instructs.",
|
||||
"dimensions": [
|
||||
{
|
||||
"name": "ai_suspicion",
|
||||
"type": "on_axis",
|
||||
"description": "If you encountered this text on a public blog, forum, or documentation site, how likely would you be to suspect it was written by an AI? Rate the overall impression, not whether specific patterns are present.",
|
||||
"levels": {
|
||||
"1.0": "Immediately obvious AI. The overall rhythm, vocabulary, and structure scream generated text.",
|
||||
"2.0": "Probably AI. Several things feel off in a way that triggers suspicion.",
|
||||
"3.0": "Could go either way. Some passages feel human, others feel generated.",
|
||||
"4.0": "Probably human. Reads naturally; would not trigger suspicion in most readers.",
|
||||
"5.0": "Certainly human. Distinctive voice, specific details, natural rhythm. No suspicion."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "information_density",
|
||||
"type": "on_axis",
|
||||
"description": "How much actual, specific information does the text convey per word? Is every sentence earning its place, or is there framing, padding, and restatement?",
|
||||
"levels": {
|
||||
"1.0": "Almost entirely padding. Significance framing, restated context, filler. Could be 80% shorter without losing information.",
|
||||
"2.0": "Mostly framing with some content. Reader has to skim past throat-clearing to find the point.",
|
||||
"3.0": "Moderate density. Some unnecessary framing but the core content is there.",
|
||||
"4.0": "Good density. Most sentences add information. Minimal padding.",
|
||||
"5.0": "Maximum density. Every sentence carries new information. Nothing could be cut without losing meaning."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "voice",
|
||||
"type": "on_axis",
|
||||
"description": "Does the text sound like a specific person wrote it, with their own perspective? Or does it sound like generic content that any competent writer could have produced?",
|
||||
"levels": {
|
||||
"1.0": "Completely generic. Could have been written by any writer. No personality, no perspective.",
|
||||
"2.0": "Mostly generic. A few moments of personality but largely indistinguishable from template writing.",
|
||||
"3.0": "Some voice present. The writer's perspective shows in places but is inconsistent.",
|
||||
"4.0": "Clear voice. The writing sounds like a particular person with a particular take.",
|
||||
"5.0": "Distinctive voice. Unmistakably one person's writing. Specific opinions, rhythm, and personality throughout."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "concreteness",
|
||||
"type": "on_axis",
|
||||
"description": "Does the text use specific, concrete details, examples, and grounded references? Or does it stay at an abstract, generalized level?",
|
||||
"levels": {
|
||||
"1.0": "Entirely abstract. General statements with no grounding in specifics.",
|
||||
"2.0": "Mostly abstract with occasional specific details.",
|
||||
"3.0": "Mix of abstract and concrete. Some specific details but they feel generic or obligatory.",
|
||||
"4.0": "Mostly concrete. Specific details that ground the writing in reality.",
|
||||
"5.0": "Vividly concrete. Details that make the writing feel lived-in and real, not assembled."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "task_completion",
|
||||
"type": "off_axis",
|
||||
"description": "Did the response actually accomplish what was asked? Did it address the task fully and correctly?",
|
||||
"levels": {
|
||||
"1.0": "Missed the point entirely. Did not accomplish the task.",
|
||||
"2.0": "Partially addressed the task but missed key aspects.",
|
||||
"3.0": "Adequately addressed the task. Nothing major missing.",
|
||||
"4.0": "Well-executed. Addressed all aspects of the task.",
|
||||
"5.0": "Excellent. Fully addressed the task with insight and appropriate depth."
|
||||
}
|
||||
},
|
||||
{
|
||||
"name": "register",
|
||||
"type": "off_axis",
|
||||
"description": "Is the tone, formality level, and style appropriate for the context and audience implied by the task?",
|
||||
"levels": {
|
||||
"1.0": "Completely wrong register. Tone is mismatched to the context.",
|
||||
"2.0": "Noticeably off. Too formal, too casual, or wrong style for the situation.",
|
||||
"3.0": "Acceptable but not ideal. Slightly mismatched in places.",
|
||||
"4.0": "Well-matched. The tone fits the context.",
|
||||
"5.0": "Perfectly calibrated. The register is exactly right for the audience and situation."
|
||||
}
|
||||
}
|
||||
],
|
||||
"score_formula": "on_axis_mean = mean(ai_suspicion, information_density, voice, concreteness); off_axis_penalty = (5 - mean(task_completion, register)) / 5; score = on_axis_mean * (1 - off_axis_penalty)"
|
||||
}
|
||||
Reference in New Issue
Block a user