v2 chunk 4: revert validator carve-out + expand publisher exceptions

Two follow-ups based on the chunk 3 verification re-run findings.

src/cmos/runtime_validator.py — revert chunk 3 fix #3 (carve-out)
  Chunk 3 fix #3 added a carve-out so shortened-form notes ("Rosen,
  7.", "Ettarh.") would pass the validator without italics, saving
  retry cost. Chunk 3 verification on the Anti-Communist draft
  showed the carve-out also let GPT-5's variance produce
  under-italicized variants of shortened forms WITH short titles
  (e.g., "A Restudy, 73." instead of the more CMOS-correct
  "*A Restudy*, 73."). The user explicitly chose accuracy over
  the cost saving and asked for the carve-out to be reverted.

  The strict italics check now applies uniformly. Simple shortened
  forms (Mitchell, 197., Ettarh.) get retried unnecessarily and
  waste API cost without producing better output. Shortened forms
  with short titles get a fair shot at the italicized version on
  the retry. Cost ↑, accuracy ↑.

  Removes _SHORTENED_FORM_RE, _looks_like_shortened_form, and the
  carve-out check from validate(). Updates module docstring with
  history note. Inverts the 2 carve-out tests in
  test_runtime_validator.py to assert shortened forms now FAIL
  the strict check, documenting the design intent for future
  reviewers.

src/cmos/note_formatter.py — expand rule 18 publisher exception list
  Chunk 3 verification showed [6] Batterson getting "NYU Press"
  expanded to "New York University Press", losing the publisher's
  canonical brand. Rule 18's exception list previously only named
  MIT Press, ALA Editions, and MLA. Expanded to include NYU Press,
  Routledge, IEEE Press, ACM Press, WHO Press, plus "Pew Research
  Center" as a non-Press canonical example.

  Also added a "when in doubt" guidance paragraph: if the
  abbreviation contains "Press" and is widely used as the
  publisher's own branding, leave it alone. The risk of losing a
  brand name (NYU Press) is worse than the risk of leaving an
  obscure abbreviation. v2 only.

Tests: 151/151 green (was 150). The +1 is the new NYU Press
prompt-content test; the 2 inverted runtime_validator tests stayed
at the same count.

Path B: runtime_validator change crosses the v1/v2 boundary
(shared infrastructure), per the same approval that authorized
the original chunk 3 fix #3. v1's bibliography formatter is
unaffected in practice because v1 outputs always have italics.
This commit is contained in:
Mark Eaton
2026-04-11 20:32:10 -04:00
parent 553240c153
commit 83339e313c
4 changed files with 91 additions and 91 deletions
+11
View File
@@ -112,6 +112,17 @@ def test_system_prompt_mentions_month_or_season_preservation():
assert "preserve" in lower
def test_system_prompt_publisher_exception_list_includes_nyu_press():
"""Chunk 4 task 2: rule 18's publisher exception list should include
NYU Press alongside the existing MIT Press / ALA Editions / MLA.
Surfaced by [6] Batterson in the chunk 3 verification re-run, where
"NYU Press" got expanded to "New York University Press" — losing
the publisher's canonical brand name. NYU Press's official self-
presentation IS "NYU Press", not the long form.
"""
assert "NYU Press" in SYSTEM_PROMPT_NOTES
def test_system_prompt_mentions_government_reports_rule():
"""Polish fix #7: v2 should explicitly handle government documents
and institutional reports the same way v1's rule 27 does — italicize
+39 -26
View File
@@ -50,46 +50,59 @@ def test_missing_italics_anywhere_fails():
assert any("italic" in f.lower() for f in result.failures)
# --- Shortened-form note carve-out (chunk 3 fix #3, Option D) -------------
# --- Shortened-form notes — strict italics check (chunk 4 task 1 revert) --
#
# Chunk 3 fix #3 added a carve-out so that shortened-form notes like
# "Rosen, 7." and "Ettarh." passed validation despite having no italics,
# avoiding retry waste on valid output. After chunk 3 verification on the
# Anti-Communist draft, the user observed that the carve-out also let
# under-italicized variants of shortened forms with short titles slip
# through (e.g., "A Restudy, 73." instead of the more CMOS-correct
# "*A Restudy*, 73.") because the model's variance produced both forms
# and the carve-out accepted the worse one.
#
# Chunk 4 task 1 reverts the carve-out: the validator's strict italics
# check now applies uniformly. The retry loop will fire on simple
# shortened forms (Mitchell, 197.) without producing better output, but
# it WILL force the model to produce italicized output on the
# shortened-form-with-short-title cases when it can. The cost goes up;
# accuracy is prioritized per the user's explicit direction.
def test_shortened_form_author_only_passes_without_italics():
"""CMOS shortened note form for an author-only reference (no page,
no title) is just the author's last name with a period: 'Ettarh.',
'Murch.', 'O'Mara.'. These contain no italics by definition.
Without a carve-out, the validator's "must have italics" rule
rejects them and the formatter's retry loop fires its full budget
on valid output. Surfaced by the Anti-Communist Formations of LIS
real-draft run; ~30 of 107 notes were correct shortened-forms
that wasted ~60 API calls retrying.
The carve-out: if a candidate has no italics AND looks like a
shortened-form note (short, name-token shape, optional page,
terminal period), the missing-italics check is skipped.
def test_shortened_form_author_only_now_fails_without_italics():
"""Post chunk-4 revert: shortened-form output without italics now
fails the validator. The retry loop will fire to give GPT-5 a chance
to italicize an embedded short title; for simple cases (single name,
no title) the retries will produce the same output and waste cost,
but for multi-token cases that DO have an italicizable title the
retries can rescue accuracy. Cost is the trade-off the user accepted.
"""
for candidate in ["Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."]:
result = validate(candidate)
assert result.passed is True, (
f"shortened-form author-only note {candidate!r} should pass "
f"validation; got failures {result.failures}"
assert result.passed is False, (
f"shortened-form note {candidate!r} should now FAIL the "
f"strict italics check (no carve-out); got passed=True"
)
assert any("italic" in f.lower() for f in result.failures)
def test_shortened_form_author_plus_page_passes_without_italics():
"""The other common CMOS shortened form is author last name + page:
'Rosen, 7.', 'Mitchell, 197.', 'CBS, 47.'. Same carve-out applies."""
def test_shortened_form_author_plus_page_now_fails_without_italics():
"""Same as above for author + page form. The retry loop will fire
on these too. For "A Restudy, 73." style entries (looks like a name
but is actually a short title) the retries can produce the
italicized variant "*A Restudy*, 73." which is more CMOS-correct.
"""
for candidate in [
"Rosen, 7.",
"Mitchell, 197.",
"CBS, 47.",
"Seybold, 282.",
"Lawrence Powell, 45.", # multi-word name
"Mitchell, 137-39.", # page range with hyphen
"Lawrence Powell, 45.",
"A Restudy, 73.", # the regression case from chunk 3 verification
]:
result = validate(candidate)
assert result.passed is True, (
f"shortened-form note {candidate!r} should pass validation; "
f"got failures {result.failures}"
assert result.passed is False, (
f"shortened-form note {candidate!r} should now FAIL the "
f"strict italics check; got passed=True"
)