Add 11 new exemplars pulled verbatim from the CMOS official quick guide
(chicagomanualofstyle.org/tools_citationguide/citation-guide-1.html):
- book_two_authors_binder: Binder & Kidder (two-author book)
- book_chapter_doyle: Doyle in Marks & Parkin (chapter in edited volume,
no page range per CMOS 18)
- book_translated_liu: Liu Xinwu, trans. Tiang (translated book, non-
Western name not comma-inverted)
- book_edition_borel: Borel, 2nd ed. via EBSCOhost (edition + database)
- book_ebook_roy: Roy, Kindle format
- journal_many_authors_snyder: 7 authors in PLOS ONE, exercises the
CMOS 18 "first 3 + et al." threshold and article-ID page format
- magazine_mead: New Yorker (tests leading-"The" drop)
- newspaper_blum: NYT with URL (tests abbreviation expansion)
- social_media_cmos_facebook: Facebook post with case preservation
- podcast_ober: Pushkin podcast with season/episode/duration
- video_cowan_ted: TED Talk with venue and duration
Every exemplar is marked canary=true — we want byte-exact regression
detection on all of them. Source citations and CMOS rule justifications
are in the TOML doc comments.
All 14 canonical outputs lint clean under LINTER_VERSION v0.2.0. The
harness will score them in the following commit.
First real loop iteration found that GPT-5 silently drops date qualifiers
("Effective", "Published", "Accessed", etc.) when reformatting web
sources. The field diff was satisfied because the canonical date field
did not include the qualifier, but the canary exact-match axis caught
the regression.
Two fixes, per the canary-upstream policy:
1. Tighten the web-page exemplar canonical: date field now includes the
"Effective" qualifier so the field diff will catch future regressions
without relying on the canary.
2. Add SYSTEM_PROMPT rule 12 instructing the formatter to preserve
semantic date qualifiers from the input.
After these fixes: scalar 1.000, canary exact-match 1.000 on all three
seed exemplars.
GPT-5 and the o-series reject temperature=0 with a 400 BadRequestError —
only the default (1) is supported for reasoning models. Add an
_is_reasoning_model helper and pass temperature only when the model name
does not start with gpt-5/o1/o3/o4. Determinism on reasoning models is a
property of the architecture, not a parameter.
Discovered on the first real LLM run against the seed exemplars.
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.
Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
sourced from chicagomanualofstyle.org quick guide
Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
cmos dev