First exercise of the tool against unfamiliar real-world drafts (user's
in-progress academic papers, not derived from CMOS quick guide
examples). Found three systemic patterns affecting ~14% of entries
across 134-entry and 13-entry drafts; this commit addresses all three.
Prompt additions:
27. GOVERNMENT DOCUMENTS / INSTITUTIONAL REPORTS — when the author is
a government body (GAO, US Census, WHO, Pew, etc.) and the source
is a standalone report, italicize the title like a book; do NOT
wrap it in quotes. CMOS 14.272.
28. PRESERVE DELIBERATE LOWERCASING of proper nouns. Examples cited in
the prompt: the journal "portal: Libraries and the Academy"
(deliberately lowercase "p"), authors bell hooks / danah boyd /
e e cummings, brand names iPhone / eBay. Headline-case
normalization must NOT touch these.
Rule 17 also extended (was newspaper/magazine only):
17. PERIODICAL NAMES — drop leading "The" from ANY periodical:
newspapers, magazines, AND scholarly journals. Write "Journal of
Academic Librarianship", not "The Journal of Academic
Librarianship"; "Library Quarterly", not "The Library Quarterly";
"American Archivist", not "The American Archivist". CMOS 14.191.
Rule explicitly excepts BOOK titles and report titles, which keep
their leading "The" (e.g., *The Library's Guide to Sexual and
Reproductive Health Information*).
Empirical impact, Reading_Disrepair (13 entries):
before: 3 errors (2 leading-The, 1 GAO format, 1 portal capitalized)
after: 0 errors
Empirical impact, HML Submission (134 entries):
before: ~21 errors (18 leading-The, 0 GAO in this draft, 3 portal)
after: ~1 error (a multi-author 2nd-author inversion bug newly
surfaced — separate fix)
Also adds scripts/analyze_draft_output.py — a triage helper that
diff-walks input and output, surfacing common failure patterns
(leading-The, doi: prefix, bare-hyphen ranges, gov-report quoted, and
intentional lowercasing stripped). False positives are tolerated since
it's a human-review tool, not a validator.
Exemplar corpus is unchanged. Re-baseline against 14 canary exemplars
under the new rules: scalar 1.000, canary 1.000 (no regressions).
User's real drafts under rough_drafts/ remain untracked — they are
in-progress academic work and don't belong in git history.
Add harness/judge.py and scripts/triage.py. The judge reads the latest
loop run from logs/runs.jsonl, finds canary failures (exact_match=False
but fields/linter passed), and asks GPT-5 to classify each as
"regression", "variant", or "unclear" with CMOS 18 section citations.
CRITICAL: verdicts are ADVISORY ONLY. They are written to
logs/triage.jsonl and never feed back into the loop scalar. Using the
judge as ground truth would let the formatter-LLM optimize against a
judge-LLM from the same model family, inviting shared-bias drift.
Design:
- harness/judge.py: Verdict dataclass, build_judge_prompt, parse_verdict
(handles code fences and normalizes unknown labels to "unclear"),
judge() with caller-injection seam matching cmos.formatter. Uses the
same _is_reasoning_model branching to skip temperature for gpt-5/o*.
Judge model is separately overridable via CMOS_JUDGE_MODEL env var
(defaults to OPENAI_MODEL, which defaults to gpt-5).
- scripts/triage.py: CLI that walks runs.jsonl, locates a target run
(default: latest), filters canary failures, calls judge on each,
appends a verdict record to logs/triage.jsonl. --dry-run available
for offline testing. Exits 0 with a note when there are no failures.
Tests: 6 new unit tests covering prompt building, JSON parsing
(including code-fence stripping and unknown-label normalization), and
caller injection. No real API calls in the test suite.
Validated on iter 3's run (canary 0.286, 10 failures):
- 8 correctly flagged as regressions, each with a cited CMOS section
(14.72, 14.76, 14.128, 14.190, 14.206, 14.212, 14.267, ...).
- 2 flagged as variants: "Kindle" vs "Kindle edition" (CMOS 14.159–
14.161 allows flexibility) and "The New Yorker" vs "New Yorker"
(CMOS 14.191 — leading "The" is optional). These surface that the
formatter's current rules 17 and 18 are stricter than CMOS strictly
requires; documenting here but not acting on yet.
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.
Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
sourced from chicagomanualofstyle.org quick guide
Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
cmos dev