Add harness/judge.py and scripts/triage.py. The judge reads the latest
loop run from logs/runs.jsonl, finds canary failures (exact_match=False
but fields/linter passed), and asks GPT-5 to classify each as
"regression", "variant", or "unclear" with CMOS 18 section citations.
CRITICAL: verdicts are ADVISORY ONLY. They are written to
logs/triage.jsonl and never feed back into the loop scalar. Using the
judge as ground truth would let the formatter-LLM optimize against a
judge-LLM from the same model family, inviting shared-bias drift.
Design:
- harness/judge.py: Verdict dataclass, build_judge_prompt, parse_verdict
(handles code fences and normalizes unknown labels to "unclear"),
judge() with caller-injection seam matching cmos.formatter. Uses the
same _is_reasoning_model branching to skip temperature for gpt-5/o*.
Judge model is separately overridable via CMOS_JUDGE_MODEL env var
(defaults to OPENAI_MODEL, which defaults to gpt-5).
- scripts/triage.py: CLI that walks runs.jsonl, locates a target run
(default: latest), filters canary failures, calls judge on each,
appends a verdict record to logs/triage.jsonl. --dry-run available
for offline testing. Exits 0 with a note when there are no failures.
Tests: 6 new unit tests covering prompt building, JSON parsing
(including code-fence stripping and unknown-label normalization), and
caller injection. No real API calls in the test suite.
Validated on iter 3's run (canary 0.286, 10 failures):
- 8 correctly flagged as regressions, each with a cited CMOS section
(14.72, 14.76, 14.128, 14.190, 14.206, 14.212, 14.267, ...).
- 2 flagged as variants: "Kindle" vs "Kindle edition" (CMOS 14.159–
14.161 allows flexibility) and "The New Yorker" vs "New Yorker"
(CMOS 14.191 — leading "The" is optional). These surface that the
formatter's current rules 17 and 18 are stricter than CMOS strictly
requires; documenting here but not acting on yet.
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.
Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
sourced from chicagomanualofstyle.org quick guide
Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
cmos dev