Mark Eaton
83339e313c
Two follow-ups based on the chunk 3 verification re-run findings. src/cmos/runtime_validator.py — revert chunk 3 fix #3 (carve-out) Chunk 3 fix #3 added a carve-out so shortened-form notes ("Rosen, 7.", "Ettarh.") would pass the validator without italics, saving retry cost. Chunk 3 verification on the Anti-Communist draft showed the carve-out also let GPT-5's variance produce under-italicized variants of shortened forms WITH short titles (e.g., "A Restudy, 73." instead of the more CMOS-correct "*A Restudy*, 73."). The user explicitly chose accuracy over the cost saving and asked for the carve-out to be reverted. The strict italics check now applies uniformly. Simple shortened forms (Mitchell, 197., Ettarh.) get retried unnecessarily and waste API cost without producing better output. Shortened forms with short titles get a fair shot at the italicized version on the retry. Cost ↑, accuracy ↑. Removes _SHORTENED_FORM_RE, _looks_like_shortened_form, and the carve-out check from validate(). Updates module docstring with history note. Inverts the 2 carve-out tests in test_runtime_validator.py to assert shortened forms now FAIL the strict check, documenting the design intent for future reviewers. src/cmos/note_formatter.py — expand rule 18 publisher exception list Chunk 3 verification showed [6] Batterson getting "NYU Press" expanded to "New York University Press", losing the publisher's canonical brand. Rule 18's exception list previously only named MIT Press, ALA Editions, and MLA. Expanded to include NYU Press, Routledge, IEEE Press, ACM Press, WHO Press, plus "Pew Research Center" as a non-Press canonical example. Also added a "when in doubt" guidance paragraph: if the abbreviation contains "Press" and is widely used as the publisher's own branding, leave it alone. The risk of losing a brand name (NYU Press) is worse than the risk of leaving an obscure abbreviation. v2 only. Tests: 151/151 green (was 150). The +1 is the new NYU Press prompt-content test; the 2 inverted runtime_validator tests stayed at the same count. Path B: runtime_validator change crosses the v1/v2 boundary (shared infrastructure), per the same approval that authorized the original chunk 3 fix #3. v1's bibliography formatter is unaffected in practice because v1 outputs always have italics.
156 lines
6.5 KiB
Python
156 lines
6.5 KiB
Python
"""Tests for src/cmos/runtime_validator.py.
|
||
|
||
Runtime validator runs at format time on a single candidate string with no
|
||
exemplar context. It checks type-independent structural sanity rules so the
|
||
formatter can self-retry when GPT-5's reasoning drift produces a malformed
|
||
entry. It is deliberately weaker than the scoring linter (which has access
|
||
to canonical fields and source type) but it catches the most common
|
||
nondeterministic failures: missing terminal period, stray Ibid., dropped
|
||
italic markers, unbalanced quotes/asterisks.
|
||
"""
|
||
|
||
from cmos.runtime_validator import validate
|
||
|
||
|
||
def test_clean_book_entry_passes():
|
||
candidate = "Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020."
|
||
result = validate(candidate)
|
||
assert result.passed is True
|
||
assert result.failures == []
|
||
|
||
|
||
def test_clean_journal_entry_passes():
|
||
candidate = (
|
||
'Kwon, Hyeyoung. "Inclusion Work: Children of Immigrants Claiming Membership in Everyday Life." '
|
||
"*American Journal of Sociology* 127, no. 6 (2022): 1818–59. "
|
||
"https://doi.org/10.1086/720277."
|
||
)
|
||
assert validate(candidate).passed
|
||
|
||
|
||
def test_missing_terminal_period_fails():
|
||
result = validate("Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020")
|
||
assert result.passed is False
|
||
assert any("period" in f.lower() for f in result.failures)
|
||
|
||
|
||
def test_ibid_fails():
|
||
result = validate("Ibid., 42.")
|
||
assert result.passed is False
|
||
assert any("ibid" in f.lower() for f in result.failures)
|
||
|
||
|
||
def test_missing_italics_anywhere_fails():
|
||
# The "New Yorker" magazine regression we saw in iter 7 variance: italic
|
||
# markers around the magazine name disappeared. This rule should catch
|
||
# that — almost every CMOS bibliography entry italicizes something.
|
||
candidate = 'Mead, Rebecca. "Terms of Aggrievement." New Yorker, December 18, 2023.'
|
||
result = validate(candidate)
|
||
assert result.passed is False
|
||
assert any("italic" in f.lower() for f in result.failures)
|
||
|
||
|
||
# --- Shortened-form notes — strict italics check (chunk 4 task 1 revert) --
|
||
#
|
||
# Chunk 3 fix #3 added a carve-out so that shortened-form notes like
|
||
# "Rosen, 7." and "Ettarh." passed validation despite having no italics,
|
||
# avoiding retry waste on valid output. After chunk 3 verification on the
|
||
# Anti-Communist draft, the user observed that the carve-out also let
|
||
# under-italicized variants of shortened forms with short titles slip
|
||
# through (e.g., "A Restudy, 73." instead of the more CMOS-correct
|
||
# "*A Restudy*, 73.") because the model's variance produced both forms
|
||
# and the carve-out accepted the worse one.
|
||
#
|
||
# Chunk 4 task 1 reverts the carve-out: the validator's strict italics
|
||
# check now applies uniformly. The retry loop will fire on simple
|
||
# shortened forms (Mitchell, 197.) without producing better output, but
|
||
# it WILL force the model to produce italicized output on the
|
||
# shortened-form-with-short-title cases when it can. The cost goes up;
|
||
# accuracy is prioritized per the user's explicit direction.
|
||
|
||
|
||
def test_shortened_form_author_only_now_fails_without_italics():
|
||
"""Post chunk-4 revert: shortened-form output without italics now
|
||
fails the validator. The retry loop will fire to give GPT-5 a chance
|
||
to italicize an embedded short title; for simple cases (single name,
|
||
no title) the retries will produce the same output and waste cost,
|
||
but for multi-token cases that DO have an italicizable title the
|
||
retries can rescue accuracy. Cost is the trade-off the user accepted.
|
||
"""
|
||
for candidate in ["Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."]:
|
||
result = validate(candidate)
|
||
assert result.passed is False, (
|
||
f"shortened-form note {candidate!r} should now FAIL the "
|
||
f"strict italics check (no carve-out); got passed=True"
|
||
)
|
||
assert any("italic" in f.lower() for f in result.failures)
|
||
|
||
|
||
def test_shortened_form_author_plus_page_now_fails_without_italics():
|
||
"""Same as above for author + page form. The retry loop will fire
|
||
on these too. For "A Restudy, 73." style entries (looks like a name
|
||
but is actually a short title) the retries can produce the
|
||
italicized variant "*A Restudy*, 73." which is more CMOS-correct.
|
||
"""
|
||
for candidate in [
|
||
"Rosen, 7.",
|
||
"Mitchell, 197.",
|
||
"CBS, 47.",
|
||
"Lawrence Powell, 45.",
|
||
"A Restudy, 73.", # the regression case from chunk 3 verification
|
||
]:
|
||
result = validate(candidate)
|
||
assert result.passed is False, (
|
||
f"shortened-form note {candidate!r} should now FAIL the "
|
||
f"strict italics check; got passed=True"
|
||
)
|
||
|
||
|
||
def test_long_no_italic_output_still_fails():
|
||
"""The carve-out must NOT be a free pass for any short string.
|
||
A long output with no italics is still likely a malformed
|
||
full-form citation (model dropped the italic title), and the
|
||
retry loop should still fire on it.
|
||
"""
|
||
candidate = (
|
||
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
|
||
"McCarthyism, Communism, and the Myth of Academic Freedom (NYU Press, 2023)."
|
||
)
|
||
result = validate(candidate)
|
||
assert result.passed is False
|
||
assert any("italic" in f.lower() for f in result.failures)
|
||
|
||
|
||
def test_short_but_unstructured_no_italic_output_still_fails():
|
||
"""A short but non-name-shaped string (e.g., a stray sentence
|
||
fragment with no name structure) should still fail. The carve-out
|
||
is NOT just "short and ends with period" — it requires the
|
||
candidate to look like a shortened-form note specifically.
|
||
"""
|
||
candidate = "this is some prose without italics."
|
||
result = validate(candidate)
|
||
assert result.passed is False, (
|
||
"lowercase prose fragment must not be accepted as a shortened-form "
|
||
"note; the carve-out should require name-token shape"
|
||
)
|
||
|
||
|
||
def test_unbalanced_italic_markers_fails():
|
||
candidate = "Yu, Charles. *Interior Chinatown. Pantheon Books, 2020."
|
||
result = validate(candidate)
|
||
assert result.passed is False
|
||
assert any("italic" in f.lower() or "*" in f for f in result.failures)
|
||
|
||
|
||
def test_unbalanced_double_quotes_fails():
|
||
candidate = 'Mead, Rebecca. "Terms of Aggrievement. *New Yorker*, December 18, 2023.'
|
||
result = validate(candidate)
|
||
assert result.passed is False
|
||
assert any("quote" in f.lower() for f in result.failures)
|
||
|
||
|
||
def test_passed_property_is_negation_of_failures():
|
||
# Sanity: passed should be True iff there are no failures.
|
||
result = validate("Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.")
|
||
assert result.passed == (result.failures == [])
|