formatter: linter-guided retry to suppress GPT-5 reasoning drift

GPT-5 is a reasoning model and is not bit-deterministic even at the API
level (no temperature override allowed). Variance testing in iter 7
showed ~1 in 6 loop runs hit a nondeterministic regression: in one run
the magazine_mead exemplar lost its italic markers around "New Yorker"
even though the exact same input had produced a clean output 5 times
prior. The scoring linter caught it (rule_magazine_name_italicized) but
the formatter still emitted the bad output to the user.

This commit adds a runtime guardrail:

- New module src/cmos/runtime_validator.py — type-independent
  structural sanity checks. Operates on a single candidate string
  with no Exemplar context, because at runtime we don't know the
  source type. Checks: ends with period, no Ibid., has at least one
  italic span (almost every CMOS bibliography entry italicizes
  something), balanced * and " markers. Deliberately weaker than the
  scoring linter; it's a fast guardrail, not a full validator.

- format_bibliography_entry now retries up to DEFAULT_MAX_RETRIES (2)
  times when the validator rejects a candidate. Independent re-calls
  are usually enough because the failures are stochastic. If every
  attempt fails, the LAST attempt is returned (no exception) — the
  caller still gets something usable, and the failure surfaces
  through the scoring linter or human review. The retry path costs
  zero on the common case (1 call per entry); ~1-2% extra calls on
  noisy drafts.

Empirical: 3 consecutive loop runs after this change are scalar 1.000
canary 1.000 (vs 5/6 clean in the variance test before). Sample is too
small to claim full suppression but the signal is positive.

Also adds a new exemplar journal_multiauthor_secondary_first_last.toml
captured from the user's HML draft (Hasanah et al., IJIDI 2024). It
exercises the case where a multi-author entry has the first author
inverted and the rest in First Last form — which the iter 7 HML run
got wrong on one entry. Variance testing showed the exemplar passes
6/6 in isolation, so the original HML failure was nondeterminism, not
a missing rule. Keeping the exemplar regardless: it adds canary
coverage of a real-world multi-author pattern, no-DOI / JSTOR-URL
variant, and the year-suffix author-date holdover stripping.

Tests: 85 → 95 (8 new for runtime_validator + 2 new for formatter
retry). All passing.
This commit is contained in:
cmos dev
2026-04-11 01:24:25 -04:00
parent a21d2614d3
commit 8b14dbc9ca
5 changed files with 240 additions and 3 deletions
@@ -0,0 +1,33 @@
# Source: derived from a real-world entry in the user's HML draft (Hasanah
# et al., IJIDI 2024). This exemplar specifically covers the CMOS rule that
# in a multi-author bibliography entry, ONLY the first author is comma-
# inverted; subsequent authors are written in "First Last" form.
#
# This is a regression test for a bug found in iter 7: the formatter saw a
# messy input where the first author was already inverted and the secondary
# authors were in First-Last form, and "fixed" the perceived inconsistency by
# inverting everyone. Per CMOS 14.72 / 14.76, only the first author is
# inverted in a bibliography entry; secondary authors stay in given-name-
# first order.
#
# Also exercises:
# - Year-suffix author-date holdover that should be stripped: "(2024). "
# - Leading "The" in journal name to be dropped (rule 17)
# - Hyphen→en-dash for page range, no elision (under 100)
# - Source provides a JSTOR URL instead of a DOI (rare variant)
source = "user draft (HML Submission); article: Hasanah et al., IJIDI 8, no. 2 (2024)"
type = "journal"
tags = ["journal", "multi-author", "no-doi", "jstor-url"]
canary = true
messy_input = "Hasanah, Anis Karunia Uswatun, Fitri Mutia, and Norhuda Salleh. (2024). “Crossing the Barriers of Library Anxiety: A Quantitative Evaluation of Indonesian Undergraduate Students with Visual Disabilities Navigating Their Academic Library.” The International Journal of Information, Diversity, & Inclusion 8, no. 2 (2024): 29-51. https://www.jstor.org/stable/48786446."
expected_bibliography = "Hasanah, Anis Karunia Uswatun, Fitri Mutia, and Norhuda Salleh. \"Crossing the Barriers of Library Anxiety: A Quantitative Evaluation of Indonesian Undergraduate Students with Visual Disabilities Navigating Their Academic Library.\" *International Journal of Information, Diversity, & Inclusion* 8, no. 2 (2024): 2951. https://www.jstor.org/stable/48786446."
[canonical]
authors = "Hasanah, Anis Karunia Uswatun, Fitri Mutia, and Norhuda Salleh"
article_title = "Crossing the Barriers of Library Anxiety: A Quantitative Evaluation of Indonesian Undergraduate Students with Visual Disabilities Navigating Their Academic Library"
journal = "International Journal of Information, Diversity, & Inclusion"
volume = 8
issue = 2
year = 2024
pages = "2951"
url = "https://www.jstor.org/stable/48786446"