Wire the v2 note formatter into the CLI so it can be invoked on
real markdown drafts. The format-notes subcommand mirrors v1's
format subcommand: parser → formatter → reassemble, with concurrent
API calls.
src/cmos/cli.py:
- Import find_notes and format_note_entry alongside the existing
v1 imports.
- Add reformat_notes(text, formatter, concurrency) that finds
pandoc-style markdown footnote definitions via find_notes,
formats each definition's text via the v2 note formatter (or
an injected fake), and substitutes the formatted text back into
the original line position. Non-definition lines preserved
byte-for-byte. Returns text unchanged when no definitions found.
- Register the format-notes argparse subparser with the same
--concurrency flag as v1's format.
- Dispatch args.command == "format-notes" to reformat_notes.
- Module docstring updated to document both subcommands.
tests/test_cli.py:
- 7 new tests for reformat_notes covering: in-place substitution,
order preservation under concurrency, prose preservation,
reference markers staying verbatim, no-op on empty input,
trailing newline preservation.
- Extended test_python_dash_m_invocation_actually_runs_main to
also assert "format-notes" appears in --help, catching accidental
subcommand removal.
Path B integrity: formatter.py, note_formatter.py, parser.py,
linter.py, harness/score.py all unchanged. No LINTER_VERSION
bump. 96/96 v1 tests still passing. Total suite: 123/123.
Real-API end-to-end smoke test on a temp draft with 2 footnote
definitions: both reformatted byte-for-byte, prose and headings
preserved, ## Conclusion section after the notes preserved.
Two prompt edits to SYSTEM_PROMPT_NOTES driven by chunk 1 smoke test
gaps on chapter_first_doyle.toml.
Rule 6 (chapter form) expanded with an explicit invariance statement:
"ed." is the canonical abbreviation regardless of editor count — do
NOT pluralize to "eds." for multiple editors. CMOS NB treats it as
an invariant abbreviation, not a number-agreeing word.
New rule 18 (publisher expansion) mirrors v1 formatter.py rule 14:
publisher names must be in full canonical form, with note-form-
specific examples ("U of Chicago Press" → "University of Chicago
Press"). Includes the same MIT Press / ALA Editions / MLA carve-out
for publishers whose canonical self-presentation legitimately uses
initials.
Two new prompt-content unit tests added to tests/test_note_formatter.py
following the v1 test_formatter.py discipline. TDD cycle: red-green-
verified end-to-end.
Real-API smoke test, 3 v2 exemplars × 2 runs each: 6/6 byte-perfect
matches (was 4/6 in chunk 1; chapter_first_doyle went 0/2 → 2/2,
book and journal still 2/2). v1 untouched, 96/96 v1 tests still
passing. Total suite: 116/116.
Begin v2 (in-text citation note form) as a parallel artifact to the
v1 bibliography formatter, per the Path B architectural decision:
no shared mutable state, no shared prompt content, no v1 changes.
New artifacts:
- src/cmos/parser.py: add find_notes() / NoteDefinition /
NotesParseResult as siblings to split_bibliography(). Targets
pandoc-style markdown footnote definitions [^marker]: text.
Single-line only; multi-line continuation deferred.
- src/cmos/note_formatter.py: new module mirroring formatter.py.
17-rule SYSTEM_PROMPT_NOTES for CMOS 18 first-occurrence note
form. Reuses cmos.runtime_validator.validate() unchanged — its
structural checks all apply to note form too. Caller injection,
retry loop, and model selection mirror v1.
- tests/test_parser.py: 7 new tests for find_notes().
- tests/test_note_formatter.py: 11 new tests mirroring v1 test
discipline (no API calls, fake-caller injection, retry semantics,
prompt smoke checks).
- exemplars/notes/: new subdirectory with 3 hand-synthesized
first-occurrence note exemplars (book, journal article, chapter
in edited book). Uses expected_note as the field name (not v1's
expected_bibliography). harness/score.py:load_exemplars uses a
non-recursive glob, so the v1 canary loader does not see these —
Path B isolation is automatic.
v1 untouched. v1 canary still loads exactly 17 exemplars. Full
test suite: 96 v1 + 18 new v2 = 114 passing.
Smoke-tested all 3 v2 exemplars against real GPT-5 (not fakes),
2 runs each. book_first_yu and journal_first_kwon: 4/4 byte-perfect
on the first try. chapter_first_doyle: 0/2, surfacing two known
prompt gaps for the next iteration:
1. Publisher abbreviation not expanded ("U of Chicago Press"
preserved instead of "University of Chicago Press"). v1's
formatter.py rule 14 is missing from the v2 prompt.
2. "ed." pluralized to "eds." for multiple editors. CMOS NB
uses "ed." invariantly regardless of editor count.
Both gaps are addressable prompt edits, not architectural problems —
exactly the kind of finding the dev loop is designed to surface.
Out of scope (deferred to chunk 2 and later): CLI extension,
shortened-form generation, document reassembly, harness scoring
loop integration, v2 linter rules, real-draft testing.
Without `if __name__ == "__main__": sys.exit(main())` at the bottom
of cli.py, `python -m cmos.cli format <path>` imports the module
but never invokes main(), so the process silently exits 0 with
empty stdout — indistinguishable from a successful run that
produced no output. Discovered during real-draft testing on
2026-04-11.
Adds a regression test that subprocesses the CLI with --help and
asserts on stdout content. argparse --help exits 0 in both broken
and fixed states; stdout content is the only discriminator.
Both invocation paths now work:
- uv run cmos format <path> (pyproject script entry)
- uv run python -m cmos.cli format <path> (module invocation)
GPT-5 is a reasoning model and is not bit-deterministic even at the API
level (no temperature override allowed). Variance testing in iter 7
showed ~1 in 6 loop runs hit a nondeterministic regression: in one run
the magazine_mead exemplar lost its italic markers around "New Yorker"
even though the exact same input had produced a clean output 5 times
prior. The scoring linter caught it (rule_magazine_name_italicized) but
the formatter still emitted the bad output to the user.
This commit adds a runtime guardrail:
- New module src/cmos/runtime_validator.py — type-independent
structural sanity checks. Operates on a single candidate string
with no Exemplar context, because at runtime we don't know the
source type. Checks: ends with period, no Ibid., has at least one
italic span (almost every CMOS bibliography entry italicizes
something), balanced * and " markers. Deliberately weaker than the
scoring linter; it's a fast guardrail, not a full validator.
- format_bibliography_entry now retries up to DEFAULT_MAX_RETRIES (2)
times when the validator rejects a candidate. Independent re-calls
are usually enough because the failures are stochastic. If every
attempt fails, the LAST attempt is returned (no exception) — the
caller still gets something usable, and the failure surfaces
through the scoring linter or human review. The retry path costs
zero on the common case (1 call per entry); ~1-2% extra calls on
noisy drafts.
Empirical: 3 consecutive loop runs after this change are scalar 1.000
canary 1.000 (vs 5/6 clean in the variance test before). Sample is too
small to claim full suppression but the signal is positive.
Also adds a new exemplar journal_multiauthor_secondary_first_last.toml
captured from the user's HML draft (Hasanah et al., IJIDI 2024). It
exercises the case where a multi-author entry has the first author
inverted and the rest in First Last form — which the iter 7 HML run
got wrong on one entry. Variance testing showed the exemplar passes
6/6 in isolation, so the original HML failure was nondeterminism, not
a missing rule. Keeping the exemplar regardless: it adds canary
coverage of a real-world multi-author pattern, no-DOI / JSTOR-URL
variant, and the year-suffix author-date holdover stripping.
Tests: 85 → 95 (8 new for runtime_validator + 2 new for formatter
retry). All passing.
reformat_draft now uses concurrent.futures.ThreadPoolExecutor with a
default of 8 workers. OpenAI SDK calls are synchronous but network-
bound, so threads release the GIL during I/O and give real speedup.
ThreadPoolExecutor.map preserves input order regardless of completion
order, so output is deterministic.
Empirical: HML draft (134 entries) went from ~25 min serial to 1m55s
with concurrency=12. ~12x speedup; further increases hit OpenAI rate
limits.
CLI gains a --concurrency flag (default 8) for tuning per draft size /
rate limit headroom. concurrency=1 forces serial execution for
debugging. New test asserts that order is preserved when concurrent
calls finish out of input order (uses a sleep-by-index fake formatter).
Add harness/judge.py and scripts/triage.py. The judge reads the latest
loop run from logs/runs.jsonl, finds canary failures (exact_match=False
but fields/linter passed), and asks GPT-5 to classify each as
"regression", "variant", or "unclear" with CMOS 18 section citations.
CRITICAL: verdicts are ADVISORY ONLY. They are written to
logs/triage.jsonl and never feed back into the loop scalar. Using the
judge as ground truth would let the formatter-LLM optimize against a
judge-LLM from the same model family, inviting shared-bias drift.
Design:
- harness/judge.py: Verdict dataclass, build_judge_prompt, parse_verdict
(handles code fences and normalizes unknown labels to "unclear"),
judge() with caller-injection seam matching cmos.formatter. Uses the
same _is_reasoning_model branching to skip temperature for gpt-5/o*.
Judge model is separately overridable via CMOS_JUDGE_MODEL env var
(defaults to OPENAI_MODEL, which defaults to gpt-5).
- scripts/triage.py: CLI that walks runs.jsonl, locates a target run
(default: latest), filters canary failures, calls judge on each,
appends a verdict record to logs/triage.jsonl. --dry-run available
for offline testing. Exits 0 with a note when there are no failures.
Tests: 6 new unit tests covering prompt building, JSON parsing
(including code-fence stripping and unknown-label normalization), and
caller injection. No real API calls in the test suite.
Validated on iter 3's run (canary 0.286, 10 failures):
- 8 correctly flagged as regressions, each with a cited CMOS section
(14.72, 14.76, 14.128, 14.190, 14.206, 14.212, 14.267, ...).
- 2 flagged as variants: "Kindle" vs "Kindle edition" (CMOS 14.159–
14.161 allows flexibility) and "The New Yorker" vs "New Yorker"
(CMOS 14.191 — leading "The" is optional). These surface that the
formatter's current rules 17 and 18 are stricter than CMOS strictly
requires; documenting here but not acting on yet.
Close the defense-in-depth gap flagged after iter 6: chapter, magazine,
newspaper, social_media, podcast, and video exemplars had ZERO applicable
linter rules, so their 100% linter pass rate was trivially true. Every
exemplar now has 3-6 structural rules firing.
Bump LINTER_VERSION to v0.3.0. Per harness discipline, this invalidates
prior run logs' scores for cross-version comparison (they stay in the
log as history). The re-baseline run still scores scalar 1.000, canary
1.000 on all 14 exemplars — formatter was already producing output that
matches the new structural conventions, so tightening the linter didn't
surface any regressions.
New rules (12):
- chapter_book_title_italicized
- chapter_in_book_marker
- magazine_name_italicized
- newspaper_name_italicized
- periodical_comma_before_date (magazine + newspaper; not journal)
- social_media_post_quoted
- social_media_platform_comma_date
- podcast_series_italicized
- podcast_episode_quoted
- podcast_format_label
- video_title_quoted (in quotes, NOT italicized)
- video_format_label
Extended:
- article_title_quoted now covers journal, magazine, newspaper,
and accepts ?/! as terminal punctuation
Each rule has a positive + negative unit test with the CMOS/Purdue source
cited in the docstring. Tests: 48 → 78 (30 new).
Per-exemplar applicable-rule counts after this change:
book 4 journal 5-6 web 3
chapter 4 magazine 5 social 4
podcast 5 video 4 newspaper 5
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.
Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
sourced from chicagomanualofstyle.org quick guide
Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
Mark Eaton