v2 chunk 4: revert validator carve-out + expand publisher exceptions
Two follow-ups based on the chunk 3 verification re-run findings. src/cmos/runtime_validator.py — revert chunk 3 fix #3 (carve-out) Chunk 3 fix #3 added a carve-out so shortened-form notes ("Rosen, 7.", "Ettarh.") would pass the validator without italics, saving retry cost. Chunk 3 verification on the Anti-Communist draft showed the carve-out also let GPT-5's variance produce under-italicized variants of shortened forms WITH short titles (e.g., "A Restudy, 73." instead of the more CMOS-correct "*A Restudy*, 73."). The user explicitly chose accuracy over the cost saving and asked for the carve-out to be reverted. The strict italics check now applies uniformly. Simple shortened forms (Mitchell, 197., Ettarh.) get retried unnecessarily and waste API cost without producing better output. Shortened forms with short titles get a fair shot at the italicized version on the retry. Cost ↑, accuracy ↑. Removes _SHORTENED_FORM_RE, _looks_like_shortened_form, and the carve-out check from validate(). Updates module docstring with history note. Inverts the 2 carve-out tests in test_runtime_validator.py to assert shortened forms now FAIL the strict check, documenting the design intent for future reviewers. src/cmos/note_formatter.py — expand rule 18 publisher exception list Chunk 3 verification showed [6] Batterson getting "NYU Press" expanded to "New York University Press", losing the publisher's canonical brand. Rule 18's exception list previously only named MIT Press, ALA Editions, and MLA. Expanded to include NYU Press, Routledge, IEEE Press, ACM Press, WHO Press, plus "Pew Research Center" as a non-Press canonical example. Also added a "when in doubt" guidance paragraph: if the abbreviation contains "Press" and is widely used as the publisher's own branding, leave it alone. The risk of losing a brand name (NYU Press) is worse than the risk of leaving an obscure abbreviation. v2 only. Tests: 151/151 green (was 150). The +1 is the new NYU Press prompt-content test; the 2 inverted runtime_validator tests stayed at the same count. Path B: runtime_validator change crosses the v1/v2 boundary (shared infrastructure), per the same approval that authorized the original chunk 3 fix #3. v1's bibliography formatter is unaffected in practice because v1 outputs always have italics.
This commit is contained in:
Mark Eaton
@@ -293,17 +293,37 @@ Rules you must follow (CMOS 18th edition specifically, NOTE form):
|
||||
|
||||
18. PUBLISHER NAMES must be in FULL canonical form. Do NOT abbreviate.
|
||||
If the messy input contains an abbreviated form, EXPAND it to the
|
||||
canonical full name. Examples:
|
||||
canonical full name. Examples of expansion:
|
||||
- "U of Chicago Press" → "University of Chicago Press"
|
||||
- "OUP" → "Oxford University Press"
|
||||
- "CUP" → "Cambridge University Press"
|
||||
- "HMC" → "Houghton Mifflin Company"
|
||||
- "Random House" stays "Random House" (already canonical)
|
||||
- "Pantheon Books" stays "Pantheon Books" (already canonical)
|
||||
The exception: publishers whose canonical self-presentation
|
||||
legitimately uses initials (e.g., "MIT Press", "ALA Editions",
|
||||
"MLA") stay in that form. When in doubt, prefer the longer
|
||||
form over the abbreviation.
|
||||
|
||||
EXCEPTION — publishers whose CANONICAL self-presentation IS the
|
||||
short form (initials or short brand) stay in that form. Do NOT
|
||||
expand these. The publisher's own website / imprint colophon is
|
||||
the source of truth here. Known cases (not exhaustive):
|
||||
- "MIT Press" (NOT "Massachusetts Institute of Technology Press")
|
||||
- "NYU Press" (NOT "New York University Press")
|
||||
- "Routledge" (already short and canonical)
|
||||
- "Sage" / "SAGE Publications" (depending on the imprint's own
|
||||
branding; do not over-expand)
|
||||
- "ALA Editions" (American Library Association's imprint)
|
||||
- "MLA" / "Modern Language Association"
|
||||
- "IEEE Press"
|
||||
- "ACM Press"
|
||||
- "WHO Press" (World Health Organization's imprint)
|
||||
- "Pew Research Center" (already canonical, no Press suffix)
|
||||
|
||||
When in doubt: if the abbreviated form contains the word "Press"
|
||||
and is widely used in academic citations as the publisher's
|
||||
own branding, leave it alone. The risk of expanding "NYU Press"
|
||||
to "New York University Press" — losing the brand the publisher
|
||||
actually uses — is worse than the risk of leaving an obscure
|
||||
abbreviation unchanged. Prefer leaving short canonical brand
|
||||
names alone over forcing expansion.
|
||||
|
||||
19. GOVERNMENT DOCUMENTS AND INSTITUTIONAL REPORTS — CMOS 14.272.
|
||||
When the author is a government body, agency, institution, or
|
||||
|
||||
@@ -23,24 +23,21 @@ bibliography entries (and CMOS 18 first-occurrence note entries):
|
||||
- Italic markers (``*``) are balanced (even count).
|
||||
- Straight double quotes (``"``) are balanced (even count).
|
||||
|
||||
**Shortened-form note carve-out (chunk 3 fix #3, Option D):** the
|
||||
"must contain italics" check is SKIPPED when the candidate looks like
|
||||
a CMOS shortened-form note — author last name (or "Lastname, page")
|
||||
with no title and no italic content. Without this carve-out, the
|
||||
validator rejects valid shortened forms ("Rosen, 7.", "Ettarh.",
|
||||
"Mitchell, 197.") and the formatter's retry loop fires its full budget
|
||||
on correct output. Surfaced by the Anti-Communist Formations of LIS
|
||||
real-draft run where ~30 of 107 notes were valid shortened forms that
|
||||
wasted ~60 API calls between them.
|
||||
|
||||
The carve-out is conservative: it only matches one or two name-token
|
||||
forms (capitalized, allowing apostrophes and hyphens for names like
|
||||
"O'Mara" and "Burden-Stelly") optionally followed by a comma and a
|
||||
page number / page range, ending in a period. Anything more complex
|
||||
(quoted short titles, italicized short titles, longer prose) still
|
||||
falls under the strict "must have italics" check. The goal is to
|
||||
catch the most common shortened-form cases without false positives
|
||||
on actual full-form citations that happen to be short.
|
||||
**History note (chunk 3 fix #3 → chunk 4 task 1 revert):** an earlier
|
||||
iteration added a carve-out so that the strict italics check was
|
||||
skipped when the candidate looked like a shortened-form note (author
|
||||
last name with optional page, no title). The carve-out saved retry
|
||||
cost on simple shortened forms, but it also let GPT-5 variance produce
|
||||
under-italicized variants of shortened forms WITH short titles
|
||||
(e.g., "A Restudy, 73." instead of the more CMOS-correct
|
||||
"*A Restudy*, 73."), because the carve-out matched both forms and
|
||||
the worse one slipped through. The user explicitly chose accuracy
|
||||
over the cost saving and asked for the carve-out to be reverted.
|
||||
The strict italics check now applies uniformly. Simple shortened
|
||||
forms (Mitchell, 197., Ettarh.) still get retried unnecessarily —
|
||||
the retries waste API calls without producing better output — but
|
||||
shortened forms with short titles get a fair shot at the italicized
|
||||
version on the retry.
|
||||
|
||||
This validator is intentionally weaker than ``cmos.linter`` (the
|
||||
scoring linter that needs an Exemplar). The point is to be a fast,
|
||||
@@ -51,7 +48,6 @@ because both use the same retry loop pattern. Edits affect both.
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@@ -64,44 +60,6 @@ class ValidationResult:
|
||||
return not self.failures
|
||||
|
||||
|
||||
# Shortened-form note pattern. Matches the two most common CMOS shortened
|
||||
# forms that legitimately have no italics:
|
||||
# - Author last name only: "Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."
|
||||
# - Author last name + page: "Rosen, 7.", "Mitchell, 197.", "CBS, 47."
|
||||
# "Lawrence Powell, 45." (multi-word name)
|
||||
# "Mitchell, 137-39." (page range)
|
||||
#
|
||||
# Each name token must start with an uppercase letter (rejects "this is some
|
||||
# prose without italics."). Apostrophes and hyphens are allowed inside name
|
||||
# tokens for names like "O'Mara" and "Burden-Stelly". A multi-word name is
|
||||
# space-separated capitalized tokens. The optional page is a comma followed
|
||||
# by digits, optionally with an ASCII or en-dash range. The whole thing
|
||||
# ends with a required period.
|
||||
#
|
||||
# Does NOT match shortened forms with embedded short titles (quoted or
|
||||
# italicized) — those are harder to disambiguate from real first-occurrence
|
||||
# notes and are left for the strict italics check / future iteration.
|
||||
_SHORTENED_FORM_RE = re.compile(
|
||||
r"^"
|
||||
r"[A-Z][\w'\-]*" # First name token (Lastname, possibly hyphenated/apostrophe)
|
||||
r"(?:\s+[A-Z][\w'\-]*)*" # Optional additional name tokens
|
||||
r"(?:,\s*\d+(?:[-\u2013]\d+)?)?" # Optional ", page" or ", page-range" / page–range
|
||||
r"\.$" # Required terminal period
|
||||
)
|
||||
|
||||
|
||||
def _looks_like_shortened_form(text: str) -> bool:
|
||||
"""True if ``text`` matches the conservative shortened-form pattern.
|
||||
|
||||
Used by ``validate`` to skip the "must have italics" check on
|
||||
legitimate shortened-form notes that have no italic content. Only
|
||||
catches author-only and author-plus-page shapes; more complex
|
||||
shortened forms (with quoted or italicized short titles) are not
|
||||
matched here and remain subject to the strict italic check.
|
||||
"""
|
||||
return _SHORTENED_FORM_RE.match(text.strip()) is not None
|
||||
|
||||
|
||||
def validate(candidate: str) -> ValidationResult:
|
||||
result = ValidationResult()
|
||||
text = candidate.rstrip()
|
||||
@@ -112,9 +70,7 @@ def validate(candidate: str) -> ValidationResult:
|
||||
if "ibid" in text.lower():
|
||||
result.failures.append("contains 'Ibid.' (deprecated in CMOS 18)")
|
||||
|
||||
# Strict italic check, with a carve-out for legitimate shortened-form
|
||||
# notes that have no italics by definition. See module docstring.
|
||||
if "*" not in text and not _looks_like_shortened_form(text):
|
||||
if "*" not in text:
|
||||
result.failures.append(
|
||||
"no italic span found (CMOS full citations usually italicize "
|
||||
"a book/journal/magazine/series/report title)"
|
||||
|
||||
@@ -112,6 +112,17 @@ def test_system_prompt_mentions_month_or_season_preservation():
|
||||
assert "preserve" in lower
|
||||
|
||||
|
||||
def test_system_prompt_publisher_exception_list_includes_nyu_press():
|
||||
"""Chunk 4 task 2: rule 18's publisher exception list should include
|
||||
NYU Press alongside the existing MIT Press / ALA Editions / MLA.
|
||||
Surfaced by [6] Batterson in the chunk 3 verification re-run, where
|
||||
"NYU Press" got expanded to "New York University Press" — losing
|
||||
the publisher's canonical brand name. NYU Press's official self-
|
||||
presentation IS "NYU Press", not the long form.
|
||||
"""
|
||||
assert "NYU Press" in SYSTEM_PROMPT_NOTES
|
||||
|
||||
|
||||
def test_system_prompt_mentions_government_reports_rule():
|
||||
"""Polish fix #7: v2 should explicitly handle government documents
|
||||
and institutional reports the same way v1's rule 27 does — italicize
|
||||
|
||||
@@ -50,46 +50,59 @@ def test_missing_italics_anywhere_fails():
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
# --- Shortened-form note carve-out (chunk 3 fix #3, Option D) -------------
|
||||
# --- Shortened-form notes — strict italics check (chunk 4 task 1 revert) --
|
||||
#
|
||||
# Chunk 3 fix #3 added a carve-out so that shortened-form notes like
|
||||
# "Rosen, 7." and "Ettarh." passed validation despite having no italics,
|
||||
# avoiding retry waste on valid output. After chunk 3 verification on the
|
||||
# Anti-Communist draft, the user observed that the carve-out also let
|
||||
# under-italicized variants of shortened forms with short titles slip
|
||||
# through (e.g., "A Restudy, 73." instead of the more CMOS-correct
|
||||
# "*A Restudy*, 73.") because the model's variance produced both forms
|
||||
# and the carve-out accepted the worse one.
|
||||
#
|
||||
# Chunk 4 task 1 reverts the carve-out: the validator's strict italics
|
||||
# check now applies uniformly. The retry loop will fire on simple
|
||||
# shortened forms (Mitchell, 197.) without producing better output, but
|
||||
# it WILL force the model to produce italicized output on the
|
||||
# shortened-form-with-short-title cases when it can. The cost goes up;
|
||||
# accuracy is prioritized per the user's explicit direction.
|
||||
|
||||
|
||||
def test_shortened_form_author_only_passes_without_italics():
|
||||
"""CMOS shortened note form for an author-only reference (no page,
|
||||
no title) is just the author's last name with a period: 'Ettarh.',
|
||||
'Murch.', 'O'Mara.'. These contain no italics by definition.
|
||||
Without a carve-out, the validator's "must have italics" rule
|
||||
rejects them and the formatter's retry loop fires its full budget
|
||||
on valid output. Surfaced by the Anti-Communist Formations of LIS
|
||||
real-draft run; ~30 of 107 notes were correct shortened-forms
|
||||
that wasted ~60 API calls retrying.
|
||||
|
||||
The carve-out: if a candidate has no italics AND looks like a
|
||||
shortened-form note (short, name-token shape, optional page,
|
||||
terminal period), the missing-italics check is skipped.
|
||||
def test_shortened_form_author_only_now_fails_without_italics():
|
||||
"""Post chunk-4 revert: shortened-form output without italics now
|
||||
fails the validator. The retry loop will fire to give GPT-5 a chance
|
||||
to italicize an embedded short title; for simple cases (single name,
|
||||
no title) the retries will produce the same output and waste cost,
|
||||
but for multi-token cases that DO have an italicizable title the
|
||||
retries can rescue accuracy. Cost is the trade-off the user accepted.
|
||||
"""
|
||||
for candidate in ["Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form author-only note {candidate!r} should pass "
|
||||
f"validation; got failures {result.failures}"
|
||||
assert result.passed is False, (
|
||||
f"shortened-form note {candidate!r} should now FAIL the "
|
||||
f"strict italics check (no carve-out); got passed=True"
|
||||
)
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
def test_shortened_form_author_plus_page_passes_without_italics():
|
||||
"""The other common CMOS shortened form is author last name + page:
|
||||
'Rosen, 7.', 'Mitchell, 197.', 'CBS, 47.'. Same carve-out applies."""
|
||||
def test_shortened_form_author_plus_page_now_fails_without_italics():
|
||||
"""Same as above for author + page form. The retry loop will fire
|
||||
on these too. For "A Restudy, 73." style entries (looks like a name
|
||||
but is actually a short title) the retries can produce the
|
||||
italicized variant "*A Restudy*, 73." which is more CMOS-correct.
|
||||
"""
|
||||
for candidate in [
|
||||
"Rosen, 7.",
|
||||
"Mitchell, 197.",
|
||||
"CBS, 47.",
|
||||
"Seybold, 282.",
|
||||
"Lawrence Powell, 45.", # multi-word name
|
||||
"Mitchell, 137-39.", # page range with hyphen
|
||||
"Lawrence Powell, 45.",
|
||||
"A Restudy, 73.", # the regression case from chunk 3 verification
|
||||
]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form note {candidate!r} should pass validation; "
|
||||
f"got failures {result.failures}"
|
||||
assert result.passed is False, (
|
||||
f"shortened-form note {candidate!r} should now FAIL the "
|
||||
f"strict italics check; got passed=True"
|
||||
)
|
||||
|
||||
|
||||
|
||||
Reference in New Issue
Block a user