cmos dev
1b11a03a0b
Add harness/judge.py and scripts/triage.py. The judge reads the latest loop run from logs/runs.jsonl, finds canary failures (exact_match=False but fields/linter passed), and asks GPT-5 to classify each as "regression", "variant", or "unclear" with CMOS 18 section citations. CRITICAL: verdicts are ADVISORY ONLY. They are written to logs/triage.jsonl and never feed back into the loop scalar. Using the judge as ground truth would let the formatter-LLM optimize against a judge-LLM from the same model family, inviting shared-bias drift. Design: - harness/judge.py: Verdict dataclass, build_judge_prompt, parse_verdict (handles code fences and normalizes unknown labels to "unclear"), judge() with caller-injection seam matching cmos.formatter. Uses the same _is_reasoning_model branching to skip temperature for gpt-5/o*. Judge model is separately overridable via CMOS_JUDGE_MODEL env var (defaults to OPENAI_MODEL, which defaults to gpt-5). - scripts/triage.py: CLI that walks runs.jsonl, locates a target run (default: latest), filters canary failures, calls judge on each, appends a verdict record to logs/triage.jsonl. --dry-run available for offline testing. Exits 0 with a note when there are no failures. Tests: 6 new unit tests covering prompt building, JSON parsing (including code-fence stripping and unknown-label normalization), and caller injection. No real API calls in the test suite. Validated on iter 3's run (canary 0.286, 10 failures): - 8 correctly flagged as regressions, each with a cited CMOS section (14.72, 14.76, 14.128, 14.190, 14.206, 14.212, 14.267, ...). - 2 flagged as variants: "Kindle" vs "Kindle edition" (CMOS 14.159– 14.161 allows flexibility) and "The New Yorker" vs "New Yorker" (CMOS 14.191 — leading "The" is optional). These surface that the formatter's current rules 17 and 18 are stricter than CMOS strictly requires; documenting here but not acting on yet.
cmos — Chicago Manual of Style 18 bibliography reformatter
A Python tool that reformats the bibliography section of an English-language markdown draft to CMOS 18th edition, notes-and-bibliography form.
Accuracy-first, iterative, built with a karpathy/autoresearch-style dev loop.
See program.md for the goal specification and
/home/claudecode1/.claude/plans/pure-mixing-yao.md for the implementation plan.
Quick start
uv sync
uv run pytest
uv run cmos format path/to/draft.md > out.md
Layout
src/cmos/formatter.py— the iterable artifact (edited every loop iteration).src/cmos/parser.py— extracts the bibliography section from a draft.src/cmos/linter.py— deterministic CMOS 18 rule checks (versioned).src/cmos/cli.py—cmos formatentry point.harness/score.py,harness/diff.py— frozen scoring.exemplars/— TOML test corpus.tests/— pytest suite (linter, parser, formatter, cli).rough_drafts/— user-supplied messy drafts for ad-hoc iteration.logs/— per-run scores, model ids, linter version hashes.