GPT-5 is a reasoning model and is not bit-deterministic even at the API
level (no temperature override allowed). Variance testing in iter 7
showed ~1 in 6 loop runs hit a nondeterministic regression: in one run
the magazine_mead exemplar lost its italic markers around "New Yorker"
even though the exact same input had produced a clean output 5 times
prior. The scoring linter caught it (rule_magazine_name_italicized) but
the formatter still emitted the bad output to the user.
This commit adds a runtime guardrail:
- New module src/cmos/runtime_validator.py — type-independent
structural sanity checks. Operates on a single candidate string
with no Exemplar context, because at runtime we don't know the
source type. Checks: ends with period, no Ibid., has at least one
italic span (almost every CMOS bibliography entry italicizes
something), balanced * and " markers. Deliberately weaker than the
scoring linter; it's a fast guardrail, not a full validator.
- format_bibliography_entry now retries up to DEFAULT_MAX_RETRIES (2)
times when the validator rejects a candidate. Independent re-calls
are usually enough because the failures are stochastic. If every
attempt fails, the LAST attempt is returned (no exception) — the
caller still gets something usable, and the failure surfaces
through the scoring linter or human review. The retry path costs
zero on the common case (1 call per entry); ~1-2% extra calls on
noisy drafts.
Empirical: 3 consecutive loop runs after this change are scalar 1.000
canary 1.000 (vs 5/6 clean in the variance test before). Sample is too
small to claim full suppression but the signal is positive.
Also adds a new exemplar journal_multiauthor_secondary_first_last.toml
captured from the user's HML draft (Hasanah et al., IJIDI 2024). It
exercises the case where a multi-author entry has the first author
inverted and the rest in First Last form — which the iter 7 HML run
got wrong on one entry. Variance testing showed the exemplar passes
6/6 in isolation, so the original HML failure was nondeterminism, not
a missing rule. Keeping the exemplar regardless: it adds canary
coverage of a real-world multi-author pattern, no-DOI / JSTOR-URL
variant, and the year-suffix author-date holdover stripping.
Tests: 85 → 95 (8 new for runtime_validator + 2 new for formatter
retry). All passing.
reformat_draft now uses concurrent.futures.ThreadPoolExecutor with a
default of 8 workers. OpenAI SDK calls are synchronous but network-
bound, so threads release the GIL during I/O and give real speedup.
ThreadPoolExecutor.map preserves input order regardless of completion
order, so output is deterministic.
Empirical: HML draft (134 entries) went from ~25 min serial to 1m55s
with concurrency=12. ~12x speedup; further increases hit OpenAI rate
limits.
CLI gains a --concurrency flag (default 8) for tuning per draft size /
rate limit headroom. concurrency=1 forces serial execution for
debugging. New test asserts that order is preserved when concurrent
calls finish out of input order (uses a sleep-by-index fake formatter).
Add harness/judge.py and scripts/triage.py. The judge reads the latest
loop run from logs/runs.jsonl, finds canary failures (exact_match=False
but fields/linter passed), and asks GPT-5 to classify each as
"regression", "variant", or "unclear" with CMOS 18 section citations.
CRITICAL: verdicts are ADVISORY ONLY. They are written to
logs/triage.jsonl and never feed back into the loop scalar. Using the
judge as ground truth would let the formatter-LLM optimize against a
judge-LLM from the same model family, inviting shared-bias drift.
Design:
- harness/judge.py: Verdict dataclass, build_judge_prompt, parse_verdict
(handles code fences and normalizes unknown labels to "unclear"),
judge() with caller-injection seam matching cmos.formatter. Uses the
same _is_reasoning_model branching to skip temperature for gpt-5/o*.
Judge model is separately overridable via CMOS_JUDGE_MODEL env var
(defaults to OPENAI_MODEL, which defaults to gpt-5).
- scripts/triage.py: CLI that walks runs.jsonl, locates a target run
(default: latest), filters canary failures, calls judge on each,
appends a verdict record to logs/triage.jsonl. --dry-run available
for offline testing. Exits 0 with a note when there are no failures.
Tests: 6 new unit tests covering prompt building, JSON parsing
(including code-fence stripping and unknown-label normalization), and
caller injection. No real API calls in the test suite.
Validated on iter 3's run (canary 0.286, 10 failures):
- 8 correctly flagged as regressions, each with a cited CMOS section
(14.72, 14.76, 14.128, 14.190, 14.206, 14.212, 14.267, ...).
- 2 flagged as variants: "Kindle" vs "Kindle edition" (CMOS 14.159–
14.161 allows flexibility) and "The New Yorker" vs "New Yorker"
(CMOS 14.191 — leading "The" is optional). These surface that the
formatter's current rules 17 and 18 are stricter than CMOS strictly
requires; documenting here but not acting on yet.
Close the defense-in-depth gap flagged after iter 6: chapter, magazine,
newspaper, social_media, podcast, and video exemplars had ZERO applicable
linter rules, so their 100% linter pass rate was trivially true. Every
exemplar now has 3-6 structural rules firing.
Bump LINTER_VERSION to v0.3.0. Per harness discipline, this invalidates
prior run logs' scores for cross-version comparison (they stay in the
log as history). The re-baseline run still scores scalar 1.000, canary
1.000 on all 14 exemplars — formatter was already producing output that
matches the new structural conventions, so tightening the linter didn't
surface any regressions.
New rules (12):
- chapter_book_title_italicized
- chapter_in_book_marker
- magazine_name_italicized
- newspaper_name_italicized
- periodical_comma_before_date (magazine + newspaper; not journal)
- social_media_post_quoted
- social_media_platform_comma_date
- podcast_series_italicized
- podcast_episode_quoted
- podcast_format_label
- video_title_quoted (in quotes, NOT italicized)
- video_format_label
Extended:
- article_title_quoted now covers journal, magazine, newspaper,
and accepts ?/! as terminal punctuation
Each rule has a positive + negative unit test with the CMOS/Purdue source
cited in the docstring. Tests: 48 → 78 (30 new).
Per-exemplar applicable-rule counts after this change:
book 4 journal 5-6 web 3
chapter 4 magazine 5 social 4
podcast 5 video 4 newspaper 5
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.
Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
sourced from chicagomanualofstyle.org quick guide
Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
cmos dev