Files
cmos/tests/test_note_formatter.py
T
Mark Eaton c21ca7b58e v2 chunk 5: 5 fixes from cross-draft triage (HML + Reading_Disrepair)
After chunk 4 verification on the Anti-Communist draft, ran v2
against the previously-untested ## Notes sections of HML (72 notes)
and Reading_Disrepair (25 notes) — 97 new notes for 204 total
across 3 drafts. The wider corpus surfaced 5 distinct issues that
weren't visible in the Anti-Communist data alone.

src/cmos/note_formatter.py:

1. Tighten substantive prose detector — exclude semicolon-bearing
   inputs (chunk 5 task 1). The HML draft was dominated by compound
   shortened references like:
     See Author1, "Title;" Author2, "Title;" Author3, "Title."
   17 of 72 HML notes matched this pattern, exceeded the 150-char
   threshold, and had no citation skeleton markers, so they were
   passed through verbatim by the chunk 3 detector. Compound
   references use semicolons as item separators; substantive prose
   doesn't. Adding `if ";" in stripped: return False` cleanly
   separates the two without affecting the genuine substantive
   notes (which contain none).

2. Rule 3 — distinguish comma-inside vs period-inside for
   article/chapter title closing quote based on whether MORE
   content follows the title. Surfaced by 8+ entries across drafts
   producing the wrong `"Title,".` pattern (comma inside followed
   by stray period outside) when the title is the last element of
   the entry. Rule 3 now explicitly says: comma inside when more
   content follows; period inside when title is the last element.

3. New rule 20 — preserve signal phrases verbatim. Surfaced by HML
   [69] and [70] where "For example," was stripped while "See" was
   preserved elsewhere. CMOS notes commonly open with signal
   phrases (See, See also, For example, Cf., Compare, But see,
   Contra, Accord, Quoted in) that indicate how a citation relates
   to the surrounding argument. Stripping them is data loss.

4. Rule 1 — explicit "no comma between author name and 'et al.'"
   in non-inverted note form. Surfaced by HML [69] where "Bignoli
   et al." became "Bignoli, et al." (extra comma). The comma
   before "et al." is a feature of inverted bibliography form,
   not non-inverted note form. Rule 1 now includes WRONG examples.

5. Citation skeleton URL marker — also accept bare-domain URLs
   without the https:// scheme. Surfaced by Reading_Disrepair [1]
   GAO entry which had `files.gao.gov/...` without `https://`,
   so the existing https?:// marker didn't match and the 190-char
   citation was passed through as substantive prose. New pattern:
   `\b\w{2,}(?:\.\w{2,})+/\S*` catches bare domains like
   files.gao.gov/path without false-positiving on common things
   like "e.g./" (single-char tokens excluded by {2,}).

Tests: 157/157 (was 153, +4 net new for prompt-content and regex
verification). The +4 is: compound-reference detection, genuine
substantive prose regression check, rule 3 / rule 20 / rule 1
prompt-content tests, bare-domain URL detection.

Path B: rule 18 (publishers, chunk 4) and rule 20 (signal phrases,
this commit) are v2-only. Rule 1 et al. clarification could in
principle apply to v1 too but v1 doesn't see "et al." in the same
non-inverted form. v1 untouched. 96/96 v1 tests still passing.

Real-draft validation pending — verification re-run against all
3 drafts will follow in the next step.
2026-04-11 22:10:30 -04:00

646 lines
27 KiB
Python

"""Tests for src/cmos/note_formatter.py — the v2 inner-loop iterable artifact.
Mirrors tests/test_formatter.py in shape and discipline. The note formatter
is iterated independently of the bibliography formatter (Path B); these
tests pin the stable seams (caller injection, retry loop, prompt smoke
checks) and leave LLM output shape testing to the v2 exemplar canary.
No real API calls in this file. Tests use injected fake callers.
"""
from cmos.note_formatter import (
MODEL,
SYSTEM_PROMPT_NOTES,
build_user_message,
format_note_entry,
)
def test_default_model_is_gpt5():
# Same default as v1. Override with OPENAI_MODEL=... in .env.
assert MODEL == "gpt-5"
def test_system_prompt_mentions_cmos_18_note_form():
assert "CMOS" in SYSTEM_PROMPT_NOTES
assert "18" in SYSTEM_PROMPT_NOTES
assert "NOTE" in SYSTEM_PROMPT_NOTES
def test_system_prompt_mentions_no_inversion():
# The single most distinctive difference from bibliography form: notes
# use normal-order author names ("First Last"), not inverted.
lower = SYSTEM_PROMPT_NOTES.lower()
assert "normal" in lower or "not inverted" in lower
def test_system_prompt_mentions_comma_separation():
# The second most distinctive difference: commas between elements,
# not periods.
lower = SYSTEM_PROMPT_NOTES.lower()
assert "comma" in lower
def test_system_prompt_excludes_leading_number():
# The note number prefix ("1. ", "2. ") is supplied externally; the
# formatter must not include it. The prompt must say so.
lower = SYSTEM_PROMPT_NOTES.lower()
assert "leading number" in lower or "do not include" in lower
def test_system_prompt_mentions_no_ibid():
# CMOS 18 deprecates Ibid., same as v1.
assert "Ibid" in SYSTEM_PROMPT_NOTES or "ibid" in SYSTEM_PROMPT_NOTES.lower()
def test_system_prompt_mentions_broader_deprecated_latin_set():
"""Rule 10 was rewritten in fix #2 to cover the broad CMOS 18
deprecated set, not just Ibid. The prompt must mention idem,
op. cit., loc. cit. so the LLM knows to refuse all of them
consistently — the Python guard catches pure-token cases, but
the prompt must cover the cases where a deprecated abbreviation
appears alongside other content (e.g. "Ibid., 47")."""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "idem" in lower
assert "op. cit" in lower or "op cit" in lower
assert "loc. cit" in lower or "loc cit" in lower
def test_system_prompt_rule_10_says_return_verbatim_not_empty():
"""Rule 10 must explicitly say "return verbatim" (or equivalent)
when the input contains a deprecated abbreviation the formatter
cannot resolve. The previous wording said "return the cleanest
possible full-form note from whatever information the messy input
contains" which the model interpreted as "return empty when there's
no information" — destroying the marker the user needs for review.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "verbatim" in lower
# And the prompt should explicitly forbid the empty-string output.
assert "do not return an empty" in lower or "not return empty" in lower
def test_system_prompt_mentions_publisher_full_form():
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
preserved 'U of Chicago Press' from the messy input instead of
expanding it to 'University of Chicago Press'. v1 formatter.py rule 14
covers this; the v2 prompt must too. The rule is "publisher names must
be in full canonical form, not abbreviated."
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "publisher" in lower
# Either "do not abbreviate" or "full canonical" or "full form" — any
# phrasing that conveys the prohibition.
assert (
"do not abbreviate" in lower
or "not abbreviated" in lower
or "full canonical" in lower
or "full form" in lower
)
def test_system_prompt_mentions_month_or_season_preservation():
"""Polish fix #5 (v2 mirror): preserve month/season info in scholarly
journal dates rather than collapsing to year-only. Mirror of the v1
test in tests/test_formatter.py — both prompts get the same rule.
Surfaced by entries [19] and [54] in the Anti-Communist Formations
of LIS draft where (Spring, 1993) and August 1952 were collapsed.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "month" in lower
assert "season" in lower
assert "preserve" in lower
def test_system_prompt_rule_3_distinguishes_comma_and_period_at_title_end():
"""Chunk 5 task 2: rule 3's punctuation guidance for note form must
distinguish between (a) title followed by more content (comma inside
quote) and (b) title as the last element of the entry (period inside
quote). Surfaced by 8+ entries across drafts producing the wrong
`"Title,".` pattern instead of `"Title."`.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
# Either explicit "last element" wording or both forms shown
assert "last element" in lower or "nothing follows" in lower
# Both punctuation patterns should be mentioned
assert "period inside" in lower or 'title."' in SYSTEM_PROMPT_NOTES.lower()
def test_system_prompt_mentions_signal_phrases_to_preserve():
"""Chunk 5 task 3: explicit rule that signal phrases at the start
of notes must be preserved verbatim. Surfaced by HML [69] and [70]
where "For example," was stripped while "See" was preserved
elsewhere — inconsistent behavior that loses meaningful authorial
intent.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "signal phrase" in lower
# The list should mention at least the common ones that were
# actually observed in the drafts
assert "see also" in lower
assert "for example" in lower
assert "cf." in lower or "compare" in lower
def test_system_prompt_rule_1_says_no_comma_before_et_al():
"""Chunk 5 task 4: rule 1 must explicitly say "et al." follows the
author name with NO comma in note form (non-inverted). Surfaced by
HML [69] where "Bignoli et al." became "Bignoli, et al." with a
spurious comma. CMOS 14.76: in non-inverted note form, the comma
that would precede "et al." in inverted bibliography form is NOT
used.
Test logic: the "no comma" prohibition must appear NEAR the
"et al." mention in rule 1 (within 300 chars), not just somewhere
else in the prompt (the polish #5 month/season rule also says "no
comma" but that's a different context).
"""
text = SYSTEM_PROMPT_NOTES.lower()
# Find all occurrences of "et al" and check if "no comma" is within
# 300 chars before or after each one
found_proximity = False
idx = text.find("et al")
while idx != -1:
window = text[max(0, idx - 300) : min(len(text), idx + 300)]
if "no comma" in window:
found_proximity = True
break
idx = text.find("et al", idx + 1)
# Also accept an explicit wrong-form example
has_wrong_example = (
'"Bignoli, et al."' in SYSTEM_PROMPT_NOTES
or '"Smith, et al."' in SYSTEM_PROMPT_NOTES
or 'wrong: "' in text and "et al" in text
)
assert found_proximity or has_wrong_example, (
"rule 1 must explicitly forbid the comma between author name "
"and 'et al.' in non-inverted note form. The 'no comma' "
"prohibition must appear near the 'et al.' rule, or an "
"explicit wrong-form example must be present."
)
def test_substantive_prose_detector_recognizes_domain_only_urls():
"""Chunk 5 task 5: the citation skeleton marker for URLs must also
recognize bare-domain URLs (without the https:// scheme), so that
real first-occurrence citations with bare-domain URLs aren't
false-positively passed through as substantive prose. Surfaced by
Reading_Disrepair [1] GAO entry which had files.gao.gov/... instead
of https://files.gao.gov/...
"""
# 190+ char input with no parens-year, no vol/no/pp, no DOI marker,
# no terminal page reference — but WITH a bare-domain URL.
input_with_bare_domain = (
"Government Accountability Office. Public Libraries: Many Buildings "
"Are Reported to Be in Poor Condition. GAO-26-107262. 18 December "
"2025. files.gao.gov/reports/GAO-26-107262/index.html"
)
assert len(input_with_bare_domain) > 150 # confirms it would otherwise hit the threshold
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
format_note_entry(input_with_bare_domain, caller=counting_caller)
assert call_count == 1, (
"Citation with bare-domain URL was incorrectly passed through "
"as substantive prose; the URL marker should also recognize "
"bare-domain URLs without the https:// scheme."
)
def test_system_prompt_publisher_exception_list_includes_nyu_press():
"""Chunk 4 task 2: rule 18's publisher exception list should include
NYU Press alongside the existing MIT Press / ALA Editions / MLA.
Surfaced by [6] Batterson in the chunk 3 verification re-run, where
"NYU Press" got expanded to "New York University Press" — losing
the publisher's canonical brand name. NYU Press's official self-
presentation IS "NYU Press", not the long form.
"""
assert "NYU Press" in SYSTEM_PROMPT_NOTES
def test_system_prompt_mentions_government_reports_rule():
"""Polish fix #7: v2 should explicitly handle government documents
and institutional reports the same way v1's rule 27 does — italicize
the title like a book, do NOT wrap in quotes. Implicit handling
worked in the chunk 2c real-draft test (entries [11] and [83]) but
explicit rule provides regression protection if a future prompt
iteration loses this implicit knowledge.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "government" in lower or "institutional" in lower
assert "report" in lower
# And the key behavior: treat like a book (italicize title), not a
# quoted article.
assert "italic" in lower # already in the prompt for general italics
def test_system_prompt_mentions_ed_invariant_for_multiple_editors():
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
pluralized 'ed.' to 'eds.' when there were two editors. CMOS NB form
uses 'ed.' invariantly regardless of editor count. The prompt must
explicitly state this so the model does not grammatically pluralize
by default.
"""
# Search for an explicit mention of "eds." being wrong, OR an explicit
# statement that "ed." is invariant / not pluralized.
lower = SYSTEM_PROMPT_NOTES.lower()
assert "eds." in lower or "invariant" in lower or "do not pluralize" in lower or "not pluralize" in lower
def test_build_user_message_contains_the_messy_input():
msg = build_user_message("yu, charles. interior chinatown. 2020, 45")
assert "yu, charles. interior chinatown. 2020, 45" in msg
def test_note_formatter_uses_injected_caller():
"""Same caller-injection seam as v1. Tests can substitute a fake without
touching the network."""
recorded: dict = {}
def fake_caller(system: str, user: str) -> str:
recorded["system"] = system
recorded["user"] = user
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
output = format_note_entry(
"yu, charles. interior chinatown. New York: Pantheon Books, 2020, p 45.",
caller=fake_caller,
)
assert output == "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
assert recorded["system"] == SYSTEM_PROMPT_NOTES
assert "yu, charles" in recorded["user"]
def test_note_formatter_strips_whitespace_from_caller_output():
def fake_caller(system: str, user: str) -> str:
return " Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45. \n"
output = format_note_entry("anything", caller=fake_caller)
assert output == "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
def test_note_formatter_retries_on_validator_failure():
"""If the runtime validator rejects the first attempt (e.g., italics
dropped), the formatter retries. Same retry semantics as v1."""
attempts = []
def flaky_caller(system: str, user: str) -> str:
attempts.append(len(attempts) + 1)
if len(attempts) == 1:
# First attempt: italics dropped — validator should fail.
return "Charles Yu, Interior Chinatown (Pantheon Books, 2020), 45."
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
output = format_note_entry("messy yu", caller=flaky_caller)
assert "*Interior Chinatown*" in output
assert len(attempts) == 2
def test_note_formatter_returns_last_attempt_after_max_retries():
"""If every attempt fails validation, return the last attempt and don't
loop forever. Same as v1 — the failure surfaces via scoring or human
review, not via a runtime exception."""
attempts = []
def always_bad(system: str, user: str) -> str:
attempts.append(1)
return "Charles Yu, Interior Chinatown (Pantheon Books, 2020), 45."
output = format_note_entry("messy", caller=always_bad, max_retries=2)
assert len(attempts) == 3 # 1 initial + 2 retries
assert "*" not in output
def test_format_note_entry_short_circuits_deprecated_latin_abbreviations():
"""When the input is ONLY a CMOS-18-deprecated Latin citation
abbreviation (ibid, idem, id., op. cit., loc. cit.), the formatter
must short-circuit the API and return the input verbatim. Surfaced
by note [61] in the Anti-Communist Formations of LIS draft, where
the formatter was given "Ibid." and (after retrying through the
runtime validator) returned an empty string — silently destroying
the marker that the user needs to find later for manual resolution.
The deprecated set per CMOS 18:
ibid, ibid., idem, id., op. cit., loc. cit.
plus common spelling/spacing variants (Op.Cit., loc cit, etc.).
The guard MUST be deterministic (Python-side, not LLM-side) and
MUST NOT call the API. Cost guarantee enforced via counting caller.
The whitespace of the original input is preserved verbatim so
reassembly in cmos.cli.reformat_notes is byte-exact.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
deprecated_inputs = [
"Ibid.",
"ibid",
"IBID.",
"Idem",
"idem.",
"id.",
"Id.",
"op. cit.",
"Op. Cit.",
"op.cit.",
"Op Cit",
"loc. cit.",
"Loc. Cit.",
"loc.cit.",
# With surrounding whitespace preserved verbatim
" Ibid. ",
"\tIbid.\n",
]
for inp in deprecated_inputs:
result = format_note_entry(inp, caller=counting_caller)
assert result == inp, (
f"Input {inp!r} must round-trip verbatim; got {result!r}"
)
assert call_count == 0, (
f"API caller was invoked {call_count} times for deprecated "
"Latin abbreviations — the short-circuit guard is missing or "
"not effective. Deprecated tokens must NEVER reach the API."
)
def test_format_note_entry_does_not_short_circuit_normal_citations():
"""The deprecated-token guard must NOT match real citations that
happen to start with one of the tokens. e.g. 'Ibid., 47' is a
common usage where a page number is appended; for now we let the
LLM see those (the prompt rule covers them). Pure-token-only
inputs short-circuit; anything with extra content goes to the API.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
# Real-looking inputs that contain "ibid" but are not pure tokens
not_short_circuited = [
"Ibid., 47.", # ibid + page — common usage
"Smith, Ibid., 47.",
"Title with the word ibid in it.",
"Charles Yu, Interior Chinatown.", # totally unrelated
]
for inp in not_short_circuited:
format_note_entry(inp, caller=recording_caller)
assert len(calls) == len(not_short_circuited), (
"Some non-pure-token inputs were short-circuited — guard is "
"too aggressive."
)
def test_format_note_entry_passes_through_substantive_prose_verbatim():
"""When the input is substantive prose — long discursive commentary
with no obvious citation skeleton — the formatter must return it
VERBATIM without calling the API. CMOS 14.39 explicitly allows
substantive notes (commentary, qualifications, cross-references).
The previous behavior extracted whatever citation it could find and
silently discarded the surrounding prose, which is data loss.
Surfaced by notes [8], [9], and [75] in the Anti-Communist
Formations of LIS draft. The user's explicit policy (chunk 3
fix #4): preserve ALL free text, as long as that does not break
other formatting.
Detection rule (conservative): length > 150 chars AND no obvious
citation skeleton markers (parenthesized year, URL, vol./no./pp.,
DOI, terminal page reference, terminal year). This catches the
egregious cases without false-positiving on legitimate citations,
which all have at least one skeleton marker.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# [8]-style: long discursive note about DuBois/Morehouse, no markers
note_8_style = (
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
"Morehouse's theory of the talented tenth is celebrated, missing "
"the note that he himself retracted such a theory later in his "
"life, as written about by Joy James. The violence of Jim Crow "
"becomes diminished into many monstrosities, since traditionally "
"black colleges and universities became monuments to human hope "
"through philanthropic funding."
)
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
# [9]-style: discursive about Wilder/Murch, no markers
note_9_style = (
"Craig Steven Wilder's Ebony and Ivy discusses the political "
"economy of anti-Blackness in higher education, and the influence "
"of not only capital and capitalists in the creation of "
"Historically Black Colleges and Universities and white elite "
"universities, but also enslaved labor to support the latter."
)
assert format_note_entry(note_9_style, caller=counting_caller) == note_9_style
# [75]-style: shorter cross-archive prose, no markers
note_75_style = (
"This letter appeared in multiple archives, including California "
"State Archives, Berkeley and UCLA special collections, and is "
"uploaded on UC Santa Barbara Servers."
)
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
assert call_count == 0, (
f"API caller was invoked {call_count} times for substantive prose — "
"the substantive-note pass-through guard is missing or not "
"effective. Free text must NEVER be sent to the API for "
"reformatting under this fix."
)
def test_format_note_entry_does_not_pass_through_compound_shortened_references():
"""Chunk 5 task 1: long compound shortened references (multiple
citations separated by semicolons) must NOT be treated as substantive
prose, even when they exceed the length threshold and have no
citation skeleton markers. Surfaced by the HML draft, where ~17 of
72 notes were compound references like:
See Meadowbrooke et al., "Information Behavior and HIV Testing;"
Kim and Syn, "Health Information on Facebook;" ...
These have no parenthesized year, no URL, no vol/no/pp markers, and
they're 200-400 chars long — so they matched the chunk 3 fix #4
substantive prose detector and were passed through verbatim, leaving
curly quotes and incorrect semicolon-quote ordering unfixed.
The discriminator: substantive prose doesn't typically use
semicolons as item separators. Compound references do. So adding
"no semicolons" as a third condition (alongside "long" and "no
skeleton markers") cleanly separates the two classes.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
# All of these are real-shaped HML compound references (synthesized to
# match observed patterns). Each is over the 150-char threshold and has
# no citation skeleton markers (no parenthesized year, no URL, no
# vol/no/pp). They MUST be sent to the LLM for formatting.
compound_refs = [
# 2-reference compound (smallest)
'Knapp, "Creating Safe and Inclusive Spacing;" Padrón, "Defending Queer Youth;" Pinsky and Brenner, "Threats to Accessing Sexuality Information."',
# 4-reference compound with "See" prefix
'See Dilevko and Gottlieb, "Pornography and Bibliographic Access;" Drabinski, "Queering the Catalog;" Billey et al., "Critique of RDA 9.7;" Baucom, "Archival Descriptions of LGBTQ Materials."',
# 3-reference compound, no prefix
'Ajayi and Omotayo, "Challenges of HIV/AIDS;" Kachota and Kassim, "Sexual and Reproductive Health Information-Seeking Behaviour;" Bankole and Busayo, "Reproductive Health Information Seeking Behaviour."',
]
for ref in compound_refs:
format_note_entry(ref, caller=recording_caller)
assert len(calls) == len(compound_refs), (
f"some compound references were short-circuited as substantive prose; "
f"expected {len(compound_refs)} calls, got {len(calls)}. The "
f"detector is over-matching."
)
def test_format_note_entry_still_passes_through_genuine_substantive_prose():
"""Regression test for chunk 5 task 1: tightening the detector with a
semicolon check must NOT break the genuine substantive-prose pass-
through that chunk 3 fix #4 was designed for. The Anti-Communist
notes [8], [9], and [75] (long discursive commentary, NO semicolons)
must still be preserved verbatim.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# Synthesized to match Anti-Communist note shapes — long discursive
# prose, no semicolons, no skeleton markers.
note_8_style = (
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
"Morehouse's theory of the talented tenth is celebrated, missing "
"the note that he himself retracted such a theory later in his "
"life, as written about by Joy James. The violence of Jim Crow "
"becomes diminished into many monstrosities, since traditionally "
"black colleges and universities became monuments to human hope "
"through philanthropic funding."
)
note_75_style = (
"This letter appeared in multiple archives, including California "
"State Archives, Berkeley and UCLA special collections, and is "
"uploaded on UC Santa Barbara Servers."
)
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
assert call_count == 0, (
"genuine substantive prose was incorrectly sent to the API; the "
"semicolon-tightening should not have affected these (no semicolons)."
)
def test_format_note_entry_does_not_pass_through_real_citations():
"""The substantive-prose guard must NOT match real citations that
happen to be long. The detection requires absence of citation
skeleton markers; real citations have at least one (year, URL,
vol/no/pp, DOI, terminal page). This test pins the negative cases
so the guard doesn't drift toward over-aggression.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
# Return validator-passing output so the retry loop doesn't fire
# and inflate the call count.
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
# Real first-occurrence book note (158 chars, ends with year+period)
note_6_style = (
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
"McCarthyism, Communism, and the Myth of Academic Freedom. "
"New York: NYU Press, 2023."
)
format_note_entry(note_6_style, caller=recording_caller)
# Real journal article note (URL marker)
note_44_style = (
"Lawrence W.S. Auld, \"The King Report: New Directions in Library "
"and Information Science Education.\" ACRL College & Research "
"Libraries News 48, no. 4, 1987. https://crln.acrl.org/example."
)
format_note_entry(note_44_style, caller=recording_caller)
# Real govt report (parenthesized year + URL)
note_54_style = (
"Robert D. Leigh, The California Librarian Education Survey: a "
"Report to President Robert G. Sproul of the University of "
"California. (New York: Columbia University, August 1952). "
"https://hdl.handle.net/example"
)
format_note_entry(note_54_style, caller=recording_caller)
# Real master's thesis (year at end)
note_107_style = (
"Christine Michele Curley, \"The School of the Library,\" "
"Unpublished Master Thesis, University of California Los "
"Angeles, 2017."
)
format_note_entry(note_107_style, caller=recording_caller)
assert len(calls) == 4, (
"Some real first-occurrence notes were short-circuited as "
"substantive prose — the detection is too aggressive. Each of "
"these has at least one citation skeleton marker and must be "
"sent to the API for reformatting."
)
def test_format_note_entry_returns_empty_input_verbatim_without_calling_api():
"""Empty or whitespace-only input must short-circuit the API call and
return the input verbatim. Without this guard, GPT-5 occasionally breaks
character on empty input and returns a conversational meta-reply asking
for content (e.g., "Please paste the citation entry you'd like
formatted..."). Surfaced by note [66] in the Anti-Communist Formations
of LIS draft.
The whitespace must be preserved verbatim so reassembly in
cmos.cli.reformat_notes is byte-exact on empty notes — the source's
spacing is part of the input contract, not noise to be normalized.
Critically, the API must NOT be called for empty input — the counting
caller asserts call_count == 0 to lock in the cost guarantee.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# Each whitespace flavor must round-trip verbatim and not call the API.
assert format_note_entry("", caller=counting_caller) == ""
assert format_note_entry(" ", caller=counting_caller) == " "
assert format_note_entry("\n", caller=counting_caller) == "\n"
assert format_note_entry("\t \n", caller=counting_caller) == "\t \n"
assert call_count == 0, (
"API caller was invoked on empty input — the short-circuit "
"guard is missing or not effective. Empty input must NEVER "
"reach the API."
)