v2 chunk 4: revert validator carve-out + expand publisher exceptions
Two follow-ups based on the chunk 3 verification re-run findings. src/cmos/runtime_validator.py — revert chunk 3 fix #3 (carve-out) Chunk 3 fix #3 added a carve-out so shortened-form notes ("Rosen, 7.", "Ettarh.") would pass the validator without italics, saving retry cost. Chunk 3 verification on the Anti-Communist draft showed the carve-out also let GPT-5's variance produce under-italicized variants of shortened forms WITH short titles (e.g., "A Restudy, 73." instead of the more CMOS-correct "*A Restudy*, 73."). The user explicitly chose accuracy over the cost saving and asked for the carve-out to be reverted. The strict italics check now applies uniformly. Simple shortened forms (Mitchell, 197., Ettarh.) get retried unnecessarily and waste API cost without producing better output. Shortened forms with short titles get a fair shot at the italicized version on the retry. Cost ↑, accuracy ↑. Removes _SHORTENED_FORM_RE, _looks_like_shortened_form, and the carve-out check from validate(). Updates module docstring with history note. Inverts the 2 carve-out tests in test_runtime_validator.py to assert shortened forms now FAIL the strict check, documenting the design intent for future reviewers. src/cmos/note_formatter.py — expand rule 18 publisher exception list Chunk 3 verification showed [6] Batterson getting "NYU Press" expanded to "New York University Press", losing the publisher's canonical brand. Rule 18's exception list previously only named MIT Press, ALA Editions, and MLA. Expanded to include NYU Press, Routledge, IEEE Press, ACM Press, WHO Press, plus "Pew Research Center" as a non-Press canonical example. Also added a "when in doubt" guidance paragraph: if the abbreviation contains "Press" and is widely used as the publisher's own branding, leave it alone. The risk of losing a brand name (NYU Press) is worse than the risk of leaving an obscure abbreviation. v2 only. Tests: 151/151 green (was 150). The +1 is the new NYU Press prompt-content test; the 2 inverted runtime_validator tests stayed at the same count. Path B: runtime_validator change crosses the v1/v2 boundary (shared infrastructure), per the same approval that authorized the original chunk 3 fix #3. v1's bibliography formatter is unaffected in practice because v1 outputs always have italics.
This commit is contained in:
@@ -50,46 +50,59 @@ def test_missing_italics_anywhere_fails():
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
# --- Shortened-form note carve-out (chunk 3 fix #3, Option D) -------------
|
||||
# --- Shortened-form notes — strict italics check (chunk 4 task 1 revert) --
|
||||
#
|
||||
# Chunk 3 fix #3 added a carve-out so that shortened-form notes like
|
||||
# "Rosen, 7." and "Ettarh." passed validation despite having no italics,
|
||||
# avoiding retry waste on valid output. After chunk 3 verification on the
|
||||
# Anti-Communist draft, the user observed that the carve-out also let
|
||||
# under-italicized variants of shortened forms with short titles slip
|
||||
# through (e.g., "A Restudy, 73." instead of the more CMOS-correct
|
||||
# "*A Restudy*, 73.") because the model's variance produced both forms
|
||||
# and the carve-out accepted the worse one.
|
||||
#
|
||||
# Chunk 4 task 1 reverts the carve-out: the validator's strict italics
|
||||
# check now applies uniformly. The retry loop will fire on simple
|
||||
# shortened forms (Mitchell, 197.) without producing better output, but
|
||||
# it WILL force the model to produce italicized output on the
|
||||
# shortened-form-with-short-title cases when it can. The cost goes up;
|
||||
# accuracy is prioritized per the user's explicit direction.
|
||||
|
||||
|
||||
def test_shortened_form_author_only_passes_without_italics():
|
||||
"""CMOS shortened note form for an author-only reference (no page,
|
||||
no title) is just the author's last name with a period: 'Ettarh.',
|
||||
'Murch.', 'O'Mara.'. These contain no italics by definition.
|
||||
Without a carve-out, the validator's "must have italics" rule
|
||||
rejects them and the formatter's retry loop fires its full budget
|
||||
on valid output. Surfaced by the Anti-Communist Formations of LIS
|
||||
real-draft run; ~30 of 107 notes were correct shortened-forms
|
||||
that wasted ~60 API calls retrying.
|
||||
|
||||
The carve-out: if a candidate has no italics AND looks like a
|
||||
shortened-form note (short, name-token shape, optional page,
|
||||
terminal period), the missing-italics check is skipped.
|
||||
def test_shortened_form_author_only_now_fails_without_italics():
|
||||
"""Post chunk-4 revert: shortened-form output without italics now
|
||||
fails the validator. The retry loop will fire to give GPT-5 a chance
|
||||
to italicize an embedded short title; for simple cases (single name,
|
||||
no title) the retries will produce the same output and waste cost,
|
||||
but for multi-token cases that DO have an italicizable title the
|
||||
retries can rescue accuracy. Cost is the trade-off the user accepted.
|
||||
"""
|
||||
for candidate in ["Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form author-only note {candidate!r} should pass "
|
||||
f"validation; got failures {result.failures}"
|
||||
assert result.passed is False, (
|
||||
f"shortened-form note {candidate!r} should now FAIL the "
|
||||
f"strict italics check (no carve-out); got passed=True"
|
||||
)
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
def test_shortened_form_author_plus_page_passes_without_italics():
|
||||
"""The other common CMOS shortened form is author last name + page:
|
||||
'Rosen, 7.', 'Mitchell, 197.', 'CBS, 47.'. Same carve-out applies."""
|
||||
def test_shortened_form_author_plus_page_now_fails_without_italics():
|
||||
"""Same as above for author + page form. The retry loop will fire
|
||||
on these too. For "A Restudy, 73." style entries (looks like a name
|
||||
but is actually a short title) the retries can produce the
|
||||
italicized variant "*A Restudy*, 73." which is more CMOS-correct.
|
||||
"""
|
||||
for candidate in [
|
||||
"Rosen, 7.",
|
||||
"Mitchell, 197.",
|
||||
"CBS, 47.",
|
||||
"Seybold, 282.",
|
||||
"Lawrence Powell, 45.", # multi-word name
|
||||
"Mitchell, 137-39.", # page range with hyphen
|
||||
"Lawrence Powell, 45.",
|
||||
"A Restudy, 73.", # the regression case from chunk 3 verification
|
||||
]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form note {candidate!r} should pass validation; "
|
||||
f"got failures {result.failures}"
|
||||
assert result.passed is False, (
|
||||
f"shortened-form note {candidate!r} should now FAIL the "
|
||||
f"strict italics check; got passed=True"
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user