ocertat/cmos
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

v2 chunk 3: bug fixes from real-draft triage + polish

Browse Source 
Six independent fixes addressing the Anti-Communist Formations of LIS
real-draft test findings (chunk 2c). Four are bug fixes for issues
that destroyed user data or wasted API calls; two are polish quality
improvements revisited from the deferred list.

src/cmos/note_formatter.py — empty input guard (fix #1)
  Empty / whitespace-only input now short-circuits the API entirely
  and returns the input verbatim. Surfaced by note [66] in the
  Anti-Communist draft, where GPT-5 broke character on empty input
  and returned a conversational meta-reply ("Please paste the
  citation entry...") that then got substituted into the document.
  Whitespace preserved so cli.reformat_notes is byte-exact on empty
  notes. v2 only.

src/cmos/note_formatter.py — deprecated Latin guard (fix #2)
  New Python guard catches the full CMOS-18-deprecated Latin
  citation set (ibid, idem, id., op. cit., loc. cit.) and returns
  input verbatim before the API call. Rule 10 in the prompt also
  rewritten: explicitly says "return verbatim" instead of the old
  "return cleanest possible full-form note", which the model
  interpreted as "return empty when there's no information",
  silently destroying the marker. Surfaced by note [61] = "Ibid.".
  v2 only.

src/cmos/runtime_validator.py — shortened-form carve-out (fix #3)
  The "must contain italics" check is now skipped when the
  candidate looks like a CMOS shortened-form note (author last
  name, optional page, no italic content). Surfaced by ~30 of 107
  notes in the Anti-Communist draft that were correctly returned
  as shortened forms ("Rosen, 7.", "Mitchell, 197.") but rejected
  by the validator's strict italic check, firing the full retry
  budget on valid output (~60 wasted API calls per run). The
  carve-out is conservative: name-token regex + terminal period;
  doesn't match unstructured prose. Shared infrastructure — affects
  v1 and v2, no-op in v1's normal workflow because v1 outputs
  always have italics.

src/cmos/note_formatter.py — substantive prose pass-through (fix #4)
  Long discursive prose with no citation skeleton markers now
  short-circuits the API and returns the input verbatim. Surfaced
  by notes [8] (689 chars), [9] (893 chars), [75] (163 chars) in
  the Anti-Communist draft — substantive notes that the formatter
  was extracting a single citation from and silently discarding
  the surrounding commentary. CMOS 14.39 explicitly allows
  substantive notes; preserving them is the user's explicit policy
  ("preserve all free text as long as that does not break other
  formatting"). Detection: length > 150 chars AND no citation
  skeleton markers (parenthesized year, URL, vol./no./pp., DOI,
  terminal page or terminal year). Conservative — does not
  false-positive on legitimate first-occurrence notes. v2 only.

src/cmos/formatter.py + note_formatter.py — month/season preservation
  (polish #5)
  Both v1 rule 10 (journal article format) and v2 rule 5 (journal
  article note form) now explicitly instruct the model to preserve
  (Month YEAR) and (Season YEAR) parentheticals when the source
  provides them. Surfaced by entries in both Reading_Disrepair and
  Anti-Communist where (Spring, 1993) and August 1952 were silently
  collapsed to (1993) and 1952. Both forms are valid CMOS but
  month/season is more informative when the source has it. Affects
  v1 and v2 in mirror.

src/cmos/note_formatter.py — government documents rule 19 (polish #7)
  v2 now has an explicit rule mirroring v1's rule 27: government
  bodies, institutional reports, and similar standalone documents
  get italicized titles and book-form treatment, NOT quoted-article
  treatment. Includes the Anti-Communist Senate of California
  Tenth Report and a hypothetical GAO example. Implicit handling
  worked in chunk 2c, but explicit rule provides regression
  protection. v2 only.

Tests: +14 net new (across test_note_formatter.py, test_formatter.py,
test_runtime_validator.py). Total suite 150/150 (was 136 before this
chunk). All v1 tests still passing.

Path B status: formatter.py and runtime_validator.py edits cross
the v1/v2 boundary, but only by user-explicit approval per the
relevant fix discussions. The shared validator was always shared;
the v1 formatter rule 10 mirror is a small additive edit that
doesn't affect the v1 prompt's existing behavior on its existing
inputs.
This commit is contained in:
Mark Eaton
2026-04-11 20:06:59 -04:00
parent 45f51c6341
commit 553240c153
6 changed files with 673 additions and 21 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_note_formatter.py
+297
View File
@@ -53,6 +53,33 @@ def test_system_prompt_mentions_no_ibid():
assert "Ibid" in SYSTEM_PROMPT_NOTES or "ibid" in SYSTEM_PROMPT_NOTES.lower()
def test_system_prompt_mentions_broader_deprecated_latin_set():
"""Rule 10 was rewritten in fix #2 to cover the broad CMOS 18
deprecated set, not just Ibid. The prompt must mention idem,
op. cit., loc. cit. so the LLM knows to refuse all of them
consistently — the Python guard catches pure-token cases, but
the prompt must cover the cases where a deprecated abbreviation
appears alongside other content (e.g. "Ibid., 47")."""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "idem" in lower
assert "op. cit" in lower or "op cit" in lower
assert "loc. cit" in lower or "loc cit" in lower
def test_system_prompt_rule_10_says_return_verbatim_not_empty():
"""Rule 10 must explicitly say "return verbatim" (or equivalent)
when the input contains a deprecated abbreviation the formatter
cannot resolve. The previous wording said "return the cleanest
possible full-form note from whatever information the messy input
contains" which the model interpreted as "return empty when there's
no information" — destroying the marker the user needs for review.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "verbatim" in lower
# And the prompt should explicitly forbid the empty-string output.
assert "do not return an empty" in lower or "not return empty" in lower
def test_system_prompt_mentions_publisher_full_form():
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
preserved 'U of Chicago Press' from the messy input instead of
@@ -72,6 +99,35 @@ def test_system_prompt_mentions_publisher_full_form():
)
def test_system_prompt_mentions_month_or_season_preservation():
"""Polish fix #5 (v2 mirror): preserve month/season info in scholarly
journal dates rather than collapsing to year-only. Mirror of the v1
test in tests/test_formatter.py — both prompts get the same rule.
Surfaced by entries [19] and [54] in the Anti-Communist Formations
of LIS draft where (Spring, 1993) and August 1952 were collapsed.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "month" in lower
assert "season" in lower
assert "preserve" in lower
def test_system_prompt_mentions_government_reports_rule():
"""Polish fix #7: v2 should explicitly handle government documents
and institutional reports the same way v1's rule 27 does — italicize
the title like a book, do NOT wrap in quotes. Implicit handling
worked in the chunk 2c real-draft test (entries [11] and [83]) but
explicit rule provides regression protection if a future prompt
iteration loses this implicit knowledge.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "government" in lower or "institutional" in lower
assert "report" in lower
# And the key behavior: treat like a book (italicize title), not a
# quoted article.
assert "italic" in lower # already in the prompt for general italics
def test_system_prompt_mentions_ed_invariant_for_multiple_editors():
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
pluralized 'ed.' to 'eds.' when there were two editors. CMOS NB form
@@ -147,3 +203,244 @@ def test_note_formatter_returns_last_attempt_after_max_retries():
output = format_note_entry("messy", caller=always_bad, max_retries=2)
assert len(attempts) == 3 # 1 initial + 2 retries
assert "*" not in output
def test_format_note_entry_short_circuits_deprecated_latin_abbreviations():
"""When the input is ONLY a CMOS-18-deprecated Latin citation
abbreviation (ibid, idem, id., op. cit., loc. cit.), the formatter
must short-circuit the API and return the input verbatim. Surfaced
by note [61] in the Anti-Communist Formations of LIS draft, where
the formatter was given "Ibid." and (after retrying through the
runtime validator) returned an empty string — silently destroying
the marker that the user needs to find later for manual resolution.
The deprecated set per CMOS 18:
ibid, ibid., idem, id., op. cit., loc. cit.
plus common spelling/spacing variants (Op.Cit., loc cit, etc.).
The guard MUST be deterministic (Python-side, not LLM-side) and
MUST NOT call the API. Cost guarantee enforced via counting caller.
The whitespace of the original input is preserved verbatim so
reassembly in cmos.cli.reformat_notes is byte-exact.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
deprecated_inputs = [
"Ibid.",
"ibid",
"IBID.",
"Idem",
"idem.",
"id.",
"Id.",
"op. cit.",
"Op. Cit.",
"op.cit.",
"Op Cit",
"loc. cit.",
"Loc. Cit.",
"loc.cit.",
# With surrounding whitespace preserved verbatim
" Ibid. ",
"\tIbid.\n",
]
for inp in deprecated_inputs:
result = format_note_entry(inp, caller=counting_caller)
assert result == inp, (
f"Input {inp!r} must round-trip verbatim; got {result!r}"
)
assert call_count == 0, (
f"API caller was invoked {call_count} times for deprecated "
"Latin abbreviations — the short-circuit guard is missing or "
"not effective. Deprecated tokens must NEVER reach the API."
)
def test_format_note_entry_does_not_short_circuit_normal_citations():
"""The deprecated-token guard must NOT match real citations that
happen to start with one of the tokens. e.g. 'Ibid., 47' is a
common usage where a page number is appended; for now we let the
LLM see those (the prompt rule covers them). Pure-token-only
inputs short-circuit; anything with extra content goes to the API.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
# Real-looking inputs that contain "ibid" but are not pure tokens
not_short_circuited = [
"Ibid., 47.", # ibid + page — common usage
"Smith, Ibid., 47.",
"Title with the word ibid in it.",
"Charles Yu, Interior Chinatown.", # totally unrelated
]
for inp in not_short_circuited:
format_note_entry(inp, caller=recording_caller)
assert len(calls) == len(not_short_circuited), (
"Some non-pure-token inputs were short-circuited — guard is "
"too aggressive."
)
def test_format_note_entry_passes_through_substantive_prose_verbatim():
"""When the input is substantive prose — long discursive commentary
with no obvious citation skeleton — the formatter must return it
VERBATIM without calling the API. CMOS 14.39 explicitly allows
substantive notes (commentary, qualifications, cross-references).
The previous behavior extracted whatever citation it could find and
silently discarded the surrounding prose, which is data loss.
Surfaced by notes [8], [9], and [75] in the Anti-Communist
Formations of LIS draft. The user's explicit policy (chunk 3
fix #4): preserve ALL free text, as long as that does not break
other formatting.
Detection rule (conservative): length > 150 chars AND no obvious
citation skeleton markers (parenthesized year, URL, vol./no./pp.,
DOI, terminal page reference, terminal year). This catches the
egregious cases without false-positiving on legitimate citations,
which all have at least one skeleton marker.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# [8]-style: long discursive note about DuBois/Morehouse, no markers
note_8_style = (
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
"Morehouse's theory of the talented tenth is celebrated, missing "
"the note that he himself retracted such a theory later in his "
"life, as written about by Joy James. The violence of Jim Crow "
"becomes diminished into many monstrosities, since traditionally "
"black colleges and universities became monuments to human hope "
"through philanthropic funding."
)
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
# [9]-style: discursive about Wilder/Murch, no markers
note_9_style = (
"Craig Steven Wilder's Ebony and Ivy discusses the political "
"economy of anti-Blackness in higher education, and the influence "
"of not only capital and capitalists in the creation of "
"Historically Black Colleges and Universities and white elite "
"universities, but also enslaved labor to support the latter."
)
assert format_note_entry(note_9_style, caller=counting_caller) == note_9_style
# [75]-style: shorter cross-archive prose, no markers
note_75_style = (
"This letter appeared in multiple archives, including California "
"State Archives, Berkeley and UCLA special collections, and is "
"uploaded on UC Santa Barbara Servers."
)
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
assert call_count == 0, (
f"API caller was invoked {call_count} times for substantive prose — "
"the substantive-note pass-through guard is missing or not "
"effective. Free text must NEVER be sent to the API for "
"reformatting under this fix."
)
def test_format_note_entry_does_not_pass_through_real_citations():
"""The substantive-prose guard must NOT match real citations that
happen to be long. The detection requires absence of citation
skeleton markers; real citations have at least one (year, URL,
vol/no/pp, DOI, terminal page). This test pins the negative cases
so the guard doesn't drift toward over-aggression.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
# Return validator-passing output so the retry loop doesn't fire
# and inflate the call count.
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
# Real first-occurrence book note (158 chars, ends with year+period)
note_6_style = (
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
"McCarthyism, Communism, and the Myth of Academic Freedom. "
"New York: NYU Press, 2023."
)
format_note_entry(note_6_style, caller=recording_caller)
# Real journal article note (URL marker)
note_44_style = (
"Lawrence W.S. Auld, \"The King Report: New Directions in Library "
"and Information Science Education.\" ACRL College & Research "
"Libraries News 48, no. 4, 1987. https://crln.acrl.org/example."
)
format_note_entry(note_44_style, caller=recording_caller)
# Real govt report (parenthesized year + URL)
note_54_style = (
"Robert D. Leigh, The California Librarian Education Survey: a "
"Report to President Robert G. Sproul of the University of "
"California. (New York: Columbia University, August 1952). "
"https://hdl.handle.net/example"
)
format_note_entry(note_54_style, caller=recording_caller)
# Real master's thesis (year at end)
note_107_style = (
"Christine Michele Curley, \"The School of the Library,\" "
"Unpublished Master Thesis, University of California Los "
"Angeles, 2017."
)
format_note_entry(note_107_style, caller=recording_caller)
assert len(calls) == 4, (
"Some real first-occurrence notes were short-circuited as "
"substantive prose — the detection is too aggressive. Each of "
"these has at least one citation skeleton marker and must be "
"sent to the API for reformatting."
)
def test_format_note_entry_returns_empty_input_verbatim_without_calling_api():
"""Empty or whitespace-only input must short-circuit the API call and
return the input verbatim. Without this guard, GPT-5 occasionally breaks
character on empty input and returns a conversational meta-reply asking
for content (e.g., "Please paste the citation entry you'd like
formatted..."). Surfaced by note [66] in the Anti-Communist Formations
of LIS draft.
The whitespace must be preserved verbatim so reassembly in
cmos.cli.reformat_notes is byte-exact on empty notes — the source's
spacing is part of the input contract, not noise to be normalized.
Critically, the API must NOT be called for empty input — the counting
caller asserts call_count == 0 to lock in the cost guarantee.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# Each whitespace flavor must round-trip verbatim and not call the API.
assert format_note_entry("", caller=counting_caller) == ""
assert format_note_entry(" ", caller=counting_caller) == " "
assert format_note_entry("\n", caller=counting_caller) == "\n"
assert format_note_entry("\t \n", caller=counting_caller) == "\t \n"
assert call_count == 0, (
"API caller was invoked on empty input — the short-circuit "
"guard is missing or not effective. Empty input must NEVER "
"reach the API."
)