After chunk 4 verification on the Anti-Communist draft, ran v2
against the previously-untested ## Notes sections of HML (72 notes)
and Reading_Disrepair (25 notes) — 97 new notes for 204 total
across 3 drafts. The wider corpus surfaced 5 distinct issues that
weren't visible in the Anti-Communist data alone.
src/cmos/note_formatter.py:
1. Tighten substantive prose detector — exclude semicolon-bearing
inputs (chunk 5 task 1). The HML draft was dominated by compound
shortened references like:
See Author1, "Title;" Author2, "Title;" Author3, "Title."
17 of 72 HML notes matched this pattern, exceeded the 150-char
threshold, and had no citation skeleton markers, so they were
passed through verbatim by the chunk 3 detector. Compound
references use semicolons as item separators; substantive prose
doesn't. Adding `if ";" in stripped: return False` cleanly
separates the two without affecting the genuine substantive
notes (which contain none).
2. Rule 3 — distinguish comma-inside vs period-inside for
article/chapter title closing quote based on whether MORE
content follows the title. Surfaced by 8+ entries across drafts
producing the wrong `"Title,".` pattern (comma inside followed
by stray period outside) when the title is the last element of
the entry. Rule 3 now explicitly says: comma inside when more
content follows; period inside when title is the last element.
3. New rule 20 — preserve signal phrases verbatim. Surfaced by HML
[69] and [70] where "For example," was stripped while "See" was
preserved elsewhere. CMOS notes commonly open with signal
phrases (See, See also, For example, Cf., Compare, But see,
Contra, Accord, Quoted in) that indicate how a citation relates
to the surrounding argument. Stripping them is data loss.
4. Rule 1 — explicit "no comma between author name and 'et al.'"
in non-inverted note form. Surfaced by HML [69] where "Bignoli
et al." became "Bignoli, et al." (extra comma). The comma
before "et al." is a feature of inverted bibliography form,
not non-inverted note form. Rule 1 now includes WRONG examples.
5. Citation skeleton URL marker — also accept bare-domain URLs
without the https:// scheme. Surfaced by Reading_Disrepair [1]
GAO entry which had `files.gao.gov/...` without `https://`,
so the existing https?:// marker didn't match and the 190-char
citation was passed through as substantive prose. New pattern:
`\b\w{2,}(?:\.\w{2,})+/\S*` catches bare domains like
files.gao.gov/path without false-positiving on common things
like "e.g./" (single-char tokens excluded by {2,}).
Tests: 157/157 (was 153, +4 net new for prompt-content and regex
verification). The +4 is: compound-reference detection, genuine
substantive prose regression check, rule 3 / rule 20 / rule 1
prompt-content tests, bare-domain URL detection.
Path B: rule 18 (publishers, chunk 4) and rule 20 (signal phrases,
this commit) are v2-only. Rule 1 et al. clarification could in
principle apply to v1 too but v1 doesn't see "et al." in the same
non-inverted form. v1 untouched. 96/96 v1 tests still passing.
Real-draft validation pending — verification re-run against all
3 drafts will follow in the next step.
646 lines
27 KiB
Python
646 lines
27 KiB
Python
"""Tests for src/cmos/note_formatter.py — the v2 inner-loop iterable artifact.
|
|
|
|
Mirrors tests/test_formatter.py in shape and discipline. The note formatter
|
|
is iterated independently of the bibliography formatter (Path B); these
|
|
tests pin the stable seams (caller injection, retry loop, prompt smoke
|
|
checks) and leave LLM output shape testing to the v2 exemplar canary.
|
|
|
|
No real API calls in this file. Tests use injected fake callers.
|
|
"""
|
|
|
|
from cmos.note_formatter import (
|
|
MODEL,
|
|
SYSTEM_PROMPT_NOTES,
|
|
build_user_message,
|
|
format_note_entry,
|
|
)
|
|
|
|
|
|
def test_default_model_is_gpt5():
|
|
# Same default as v1. Override with OPENAI_MODEL=... in .env.
|
|
assert MODEL == "gpt-5"
|
|
|
|
|
|
def test_system_prompt_mentions_cmos_18_note_form():
|
|
assert "CMOS" in SYSTEM_PROMPT_NOTES
|
|
assert "18" in SYSTEM_PROMPT_NOTES
|
|
assert "NOTE" in SYSTEM_PROMPT_NOTES
|
|
|
|
|
|
def test_system_prompt_mentions_no_inversion():
|
|
# The single most distinctive difference from bibliography form: notes
|
|
# use normal-order author names ("First Last"), not inverted.
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "normal" in lower or "not inverted" in lower
|
|
|
|
|
|
def test_system_prompt_mentions_comma_separation():
|
|
# The second most distinctive difference: commas between elements,
|
|
# not periods.
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "comma" in lower
|
|
|
|
|
|
def test_system_prompt_excludes_leading_number():
|
|
# The note number prefix ("1. ", "2. ") is supplied externally; the
|
|
# formatter must not include it. The prompt must say so.
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "leading number" in lower or "do not include" in lower
|
|
|
|
|
|
def test_system_prompt_mentions_no_ibid():
|
|
# CMOS 18 deprecates Ibid., same as v1.
|
|
assert "Ibid" in SYSTEM_PROMPT_NOTES or "ibid" in SYSTEM_PROMPT_NOTES.lower()
|
|
|
|
|
|
def test_system_prompt_mentions_broader_deprecated_latin_set():
|
|
"""Rule 10 was rewritten in fix #2 to cover the broad CMOS 18
|
|
deprecated set, not just Ibid. The prompt must mention idem,
|
|
op. cit., loc. cit. so the LLM knows to refuse all of them
|
|
consistently — the Python guard catches pure-token cases, but
|
|
the prompt must cover the cases where a deprecated abbreviation
|
|
appears alongside other content (e.g. "Ibid., 47")."""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "idem" in lower
|
|
assert "op. cit" in lower or "op cit" in lower
|
|
assert "loc. cit" in lower or "loc cit" in lower
|
|
|
|
|
|
def test_system_prompt_rule_10_says_return_verbatim_not_empty():
|
|
"""Rule 10 must explicitly say "return verbatim" (or equivalent)
|
|
when the input contains a deprecated abbreviation the formatter
|
|
cannot resolve. The previous wording said "return the cleanest
|
|
possible full-form note from whatever information the messy input
|
|
contains" which the model interpreted as "return empty when there's
|
|
no information" — destroying the marker the user needs for review.
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "verbatim" in lower
|
|
# And the prompt should explicitly forbid the empty-string output.
|
|
assert "do not return an empty" in lower or "not return empty" in lower
|
|
|
|
|
|
def test_system_prompt_mentions_publisher_full_form():
|
|
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
|
|
preserved 'U of Chicago Press' from the messy input instead of
|
|
expanding it to 'University of Chicago Press'. v1 formatter.py rule 14
|
|
covers this; the v2 prompt must too. The rule is "publisher names must
|
|
be in full canonical form, not abbreviated."
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "publisher" in lower
|
|
# Either "do not abbreviate" or "full canonical" or "full form" — any
|
|
# phrasing that conveys the prohibition.
|
|
assert (
|
|
"do not abbreviate" in lower
|
|
or "not abbreviated" in lower
|
|
or "full canonical" in lower
|
|
or "full form" in lower
|
|
)
|
|
|
|
|
|
def test_system_prompt_mentions_month_or_season_preservation():
|
|
"""Polish fix #5 (v2 mirror): preserve month/season info in scholarly
|
|
journal dates rather than collapsing to year-only. Mirror of the v1
|
|
test in tests/test_formatter.py — both prompts get the same rule.
|
|
Surfaced by entries [19] and [54] in the Anti-Communist Formations
|
|
of LIS draft where (Spring, 1993) and August 1952 were collapsed.
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "month" in lower
|
|
assert "season" in lower
|
|
assert "preserve" in lower
|
|
|
|
|
|
def test_system_prompt_rule_3_distinguishes_comma_and_period_at_title_end():
|
|
"""Chunk 5 task 2: rule 3's punctuation guidance for note form must
|
|
distinguish between (a) title followed by more content (comma inside
|
|
quote) and (b) title as the last element of the entry (period inside
|
|
quote). Surfaced by 8+ entries across drafts producing the wrong
|
|
`"Title,".` pattern instead of `"Title."`.
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
# Either explicit "last element" wording or both forms shown
|
|
assert "last element" in lower or "nothing follows" in lower
|
|
# Both punctuation patterns should be mentioned
|
|
assert "period inside" in lower or 'title."' in SYSTEM_PROMPT_NOTES.lower()
|
|
|
|
|
|
def test_system_prompt_mentions_signal_phrases_to_preserve():
|
|
"""Chunk 5 task 3: explicit rule that signal phrases at the start
|
|
of notes must be preserved verbatim. Surfaced by HML [69] and [70]
|
|
where "For example," was stripped while "See" was preserved
|
|
elsewhere — inconsistent behavior that loses meaningful authorial
|
|
intent.
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "signal phrase" in lower
|
|
# The list should mention at least the common ones that were
|
|
# actually observed in the drafts
|
|
assert "see also" in lower
|
|
assert "for example" in lower
|
|
assert "cf." in lower or "compare" in lower
|
|
|
|
|
|
def test_system_prompt_rule_1_says_no_comma_before_et_al():
|
|
"""Chunk 5 task 4: rule 1 must explicitly say "et al." follows the
|
|
author name with NO comma in note form (non-inverted). Surfaced by
|
|
HML [69] where "Bignoli et al." became "Bignoli, et al." with a
|
|
spurious comma. CMOS 14.76: in non-inverted note form, the comma
|
|
that would precede "et al." in inverted bibliography form is NOT
|
|
used.
|
|
|
|
Test logic: the "no comma" prohibition must appear NEAR the
|
|
"et al." mention in rule 1 (within 300 chars), not just somewhere
|
|
else in the prompt (the polish #5 month/season rule also says "no
|
|
comma" but that's a different context).
|
|
"""
|
|
text = SYSTEM_PROMPT_NOTES.lower()
|
|
# Find all occurrences of "et al" and check if "no comma" is within
|
|
# 300 chars before or after each one
|
|
found_proximity = False
|
|
idx = text.find("et al")
|
|
while idx != -1:
|
|
window = text[max(0, idx - 300) : min(len(text), idx + 300)]
|
|
if "no comma" in window:
|
|
found_proximity = True
|
|
break
|
|
idx = text.find("et al", idx + 1)
|
|
|
|
# Also accept an explicit wrong-form example
|
|
has_wrong_example = (
|
|
'"Bignoli, et al."' in SYSTEM_PROMPT_NOTES
|
|
or '"Smith, et al."' in SYSTEM_PROMPT_NOTES
|
|
or 'wrong: "' in text and "et al" in text
|
|
)
|
|
|
|
assert found_proximity or has_wrong_example, (
|
|
"rule 1 must explicitly forbid the comma between author name "
|
|
"and 'et al.' in non-inverted note form. The 'no comma' "
|
|
"prohibition must appear near the 'et al.' rule, or an "
|
|
"explicit wrong-form example must be present."
|
|
)
|
|
|
|
|
|
def test_substantive_prose_detector_recognizes_domain_only_urls():
|
|
"""Chunk 5 task 5: the citation skeleton marker for URLs must also
|
|
recognize bare-domain URLs (without the https:// scheme), so that
|
|
real first-occurrence citations with bare-domain URLs aren't
|
|
false-positively passed through as substantive prose. Surfaced by
|
|
Reading_Disrepair [1] GAO entry which had files.gao.gov/... instead
|
|
of https://files.gao.gov/...
|
|
"""
|
|
# 190+ char input with no parens-year, no vol/no/pp, no DOI marker,
|
|
# no terminal page reference — but WITH a bare-domain URL.
|
|
input_with_bare_domain = (
|
|
"Government Accountability Office. Public Libraries: Many Buildings "
|
|
"Are Reported to Be in Poor Condition. GAO-26-107262. 18 December "
|
|
"2025. files.gao.gov/reports/GAO-26-107262/index.html"
|
|
)
|
|
assert len(input_with_bare_domain) > 150 # confirms it would otherwise hit the threshold
|
|
|
|
call_count = 0
|
|
|
|
def counting_caller(system: str, user: str) -> str:
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
|
|
|
|
format_note_entry(input_with_bare_domain, caller=counting_caller)
|
|
assert call_count == 1, (
|
|
"Citation with bare-domain URL was incorrectly passed through "
|
|
"as substantive prose; the URL marker should also recognize "
|
|
"bare-domain URLs without the https:// scheme."
|
|
)
|
|
|
|
|
|
def test_system_prompt_publisher_exception_list_includes_nyu_press():
|
|
"""Chunk 4 task 2: rule 18's publisher exception list should include
|
|
NYU Press alongside the existing MIT Press / ALA Editions / MLA.
|
|
Surfaced by [6] Batterson in the chunk 3 verification re-run, where
|
|
"NYU Press" got expanded to "New York University Press" — losing
|
|
the publisher's canonical brand name. NYU Press's official self-
|
|
presentation IS "NYU Press", not the long form.
|
|
"""
|
|
assert "NYU Press" in SYSTEM_PROMPT_NOTES
|
|
|
|
|
|
def test_system_prompt_mentions_government_reports_rule():
|
|
"""Polish fix #7: v2 should explicitly handle government documents
|
|
and institutional reports the same way v1's rule 27 does — italicize
|
|
the title like a book, do NOT wrap in quotes. Implicit handling
|
|
worked in the chunk 2c real-draft test (entries [11] and [83]) but
|
|
explicit rule provides regression protection if a future prompt
|
|
iteration loses this implicit knowledge.
|
|
"""
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "government" in lower or "institutional" in lower
|
|
assert "report" in lower
|
|
# And the key behavior: treat like a book (italicize title), not a
|
|
# quoted article.
|
|
assert "italic" in lower # already in the prompt for general italics
|
|
|
|
|
|
def test_system_prompt_mentions_ed_invariant_for_multiple_editors():
|
|
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
|
|
pluralized 'ed.' to 'eds.' when there were two editors. CMOS NB form
|
|
uses 'ed.' invariantly regardless of editor count. The prompt must
|
|
explicitly state this so the model does not grammatically pluralize
|
|
by default.
|
|
"""
|
|
# Search for an explicit mention of "eds." being wrong, OR an explicit
|
|
# statement that "ed." is invariant / not pluralized.
|
|
lower = SYSTEM_PROMPT_NOTES.lower()
|
|
assert "eds." in lower or "invariant" in lower or "do not pluralize" in lower or "not pluralize" in lower
|
|
|
|
|
|
def test_build_user_message_contains_the_messy_input():
|
|
msg = build_user_message("yu, charles. interior chinatown. 2020, 45")
|
|
assert "yu, charles. interior chinatown. 2020, 45" in msg
|
|
|
|
|
|
def test_note_formatter_uses_injected_caller():
|
|
"""Same caller-injection seam as v1. Tests can substitute a fake without
|
|
touching the network."""
|
|
recorded: dict = {}
|
|
|
|
def fake_caller(system: str, user: str) -> str:
|
|
recorded["system"] = system
|
|
recorded["user"] = user
|
|
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
|
|
|
output = format_note_entry(
|
|
"yu, charles. interior chinatown. New York: Pantheon Books, 2020, p 45.",
|
|
caller=fake_caller,
|
|
)
|
|
assert output == "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
|
assert recorded["system"] == SYSTEM_PROMPT_NOTES
|
|
assert "yu, charles" in recorded["user"]
|
|
|
|
|
|
def test_note_formatter_strips_whitespace_from_caller_output():
|
|
def fake_caller(system: str, user: str) -> str:
|
|
return " Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45. \n"
|
|
|
|
output = format_note_entry("anything", caller=fake_caller)
|
|
assert output == "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
|
|
|
|
|
def test_note_formatter_retries_on_validator_failure():
|
|
"""If the runtime validator rejects the first attempt (e.g., italics
|
|
dropped), the formatter retries. Same retry semantics as v1."""
|
|
attempts = []
|
|
|
|
def flaky_caller(system: str, user: str) -> str:
|
|
attempts.append(len(attempts) + 1)
|
|
if len(attempts) == 1:
|
|
# First attempt: italics dropped — validator should fail.
|
|
return "Charles Yu, Interior Chinatown (Pantheon Books, 2020), 45."
|
|
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
|
|
|
output = format_note_entry("messy yu", caller=flaky_caller)
|
|
assert "*Interior Chinatown*" in output
|
|
assert len(attempts) == 2
|
|
|
|
|
|
def test_note_formatter_returns_last_attempt_after_max_retries():
|
|
"""If every attempt fails validation, return the last attempt and don't
|
|
loop forever. Same as v1 — the failure surfaces via scoring or human
|
|
review, not via a runtime exception."""
|
|
attempts = []
|
|
|
|
def always_bad(system: str, user: str) -> str:
|
|
attempts.append(1)
|
|
return "Charles Yu, Interior Chinatown (Pantheon Books, 2020), 45."
|
|
|
|
output = format_note_entry("messy", caller=always_bad, max_retries=2)
|
|
assert len(attempts) == 3 # 1 initial + 2 retries
|
|
assert "*" not in output
|
|
|
|
|
|
def test_format_note_entry_short_circuits_deprecated_latin_abbreviations():
|
|
"""When the input is ONLY a CMOS-18-deprecated Latin citation
|
|
abbreviation (ibid, idem, id., op. cit., loc. cit.), the formatter
|
|
must short-circuit the API and return the input verbatim. Surfaced
|
|
by note [61] in the Anti-Communist Formations of LIS draft, where
|
|
the formatter was given "Ibid." and (after retrying through the
|
|
runtime validator) returned an empty string — silently destroying
|
|
the marker that the user needs to find later for manual resolution.
|
|
|
|
The deprecated set per CMOS 18:
|
|
ibid, ibid., idem, id., op. cit., loc. cit.
|
|
plus common spelling/spacing variants (Op.Cit., loc cit, etc.).
|
|
|
|
The guard MUST be deterministic (Python-side, not LLM-side) and
|
|
MUST NOT call the API. Cost guarantee enforced via counting caller.
|
|
The whitespace of the original input is preserved verbatim so
|
|
reassembly in cmos.cli.reformat_notes is byte-exact.
|
|
"""
|
|
call_count = 0
|
|
|
|
def counting_caller(system: str, user: str) -> str:
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return "should not be called"
|
|
|
|
deprecated_inputs = [
|
|
"Ibid.",
|
|
"ibid",
|
|
"IBID.",
|
|
"Idem",
|
|
"idem.",
|
|
"id.",
|
|
"Id.",
|
|
"op. cit.",
|
|
"Op. Cit.",
|
|
"op.cit.",
|
|
"Op Cit",
|
|
"loc. cit.",
|
|
"Loc. Cit.",
|
|
"loc.cit.",
|
|
# With surrounding whitespace preserved verbatim
|
|
" Ibid. ",
|
|
"\tIbid.\n",
|
|
]
|
|
for inp in deprecated_inputs:
|
|
result = format_note_entry(inp, caller=counting_caller)
|
|
assert result == inp, (
|
|
f"Input {inp!r} must round-trip verbatim; got {result!r}"
|
|
)
|
|
|
|
assert call_count == 0, (
|
|
f"API caller was invoked {call_count} times for deprecated "
|
|
"Latin abbreviations — the short-circuit guard is missing or "
|
|
"not effective. Deprecated tokens must NEVER reach the API."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_does_not_short_circuit_normal_citations():
|
|
"""The deprecated-token guard must NOT match real citations that
|
|
happen to start with one of the tokens. e.g. 'Ibid., 47' is a
|
|
common usage where a page number is appended; for now we let the
|
|
LLM see those (the prompt rule covers them). Pure-token-only
|
|
inputs short-circuit; anything with extra content goes to the API.
|
|
"""
|
|
calls: list[str] = []
|
|
|
|
def recording_caller(system: str, user: str) -> str:
|
|
calls.append(user)
|
|
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
|
|
|
# Real-looking inputs that contain "ibid" but are not pure tokens
|
|
not_short_circuited = [
|
|
"Ibid., 47.", # ibid + page — common usage
|
|
"Smith, Ibid., 47.",
|
|
"Title with the word ibid in it.",
|
|
"Charles Yu, Interior Chinatown.", # totally unrelated
|
|
]
|
|
for inp in not_short_circuited:
|
|
format_note_entry(inp, caller=recording_caller)
|
|
|
|
assert len(calls) == len(not_short_circuited), (
|
|
"Some non-pure-token inputs were short-circuited — guard is "
|
|
"too aggressive."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_passes_through_substantive_prose_verbatim():
|
|
"""When the input is substantive prose — long discursive commentary
|
|
with no obvious citation skeleton — the formatter must return it
|
|
VERBATIM without calling the API. CMOS 14.39 explicitly allows
|
|
substantive notes (commentary, qualifications, cross-references).
|
|
The previous behavior extracted whatever citation it could find and
|
|
silently discarded the surrounding prose, which is data loss.
|
|
|
|
Surfaced by notes [8], [9], and [75] in the Anti-Communist
|
|
Formations of LIS draft. The user's explicit policy (chunk 3
|
|
fix #4): preserve ALL free text, as long as that does not break
|
|
other formatting.
|
|
|
|
Detection rule (conservative): length > 150 chars AND no obvious
|
|
citation skeleton markers (parenthesized year, URL, vol./no./pp.,
|
|
DOI, terminal page reference, terminal year). This catches the
|
|
egregious cases without false-positiving on legitimate citations,
|
|
which all have at least one skeleton marker.
|
|
"""
|
|
call_count = 0
|
|
|
|
def counting_caller(system: str, user: str) -> str:
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return "should not be called"
|
|
|
|
# [8]-style: long discursive note about DuBois/Morehouse, no markers
|
|
note_8_style = (
|
|
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
|
|
"Morehouse's theory of the talented tenth is celebrated, missing "
|
|
"the note that he himself retracted such a theory later in his "
|
|
"life, as written about by Joy James. The violence of Jim Crow "
|
|
"becomes diminished into many monstrosities, since traditionally "
|
|
"black colleges and universities became monuments to human hope "
|
|
"through philanthropic funding."
|
|
)
|
|
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
|
|
|
|
# [9]-style: discursive about Wilder/Murch, no markers
|
|
note_9_style = (
|
|
"Craig Steven Wilder's Ebony and Ivy discusses the political "
|
|
"economy of anti-Blackness in higher education, and the influence "
|
|
"of not only capital and capitalists in the creation of "
|
|
"Historically Black Colleges and Universities and white elite "
|
|
"universities, but also enslaved labor to support the latter."
|
|
)
|
|
assert format_note_entry(note_9_style, caller=counting_caller) == note_9_style
|
|
|
|
# [75]-style: shorter cross-archive prose, no markers
|
|
note_75_style = (
|
|
"This letter appeared in multiple archives, including California "
|
|
"State Archives, Berkeley and UCLA special collections, and is "
|
|
"uploaded on UC Santa Barbara Servers."
|
|
)
|
|
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
|
|
|
|
assert call_count == 0, (
|
|
f"API caller was invoked {call_count} times for substantive prose — "
|
|
"the substantive-note pass-through guard is missing or not "
|
|
"effective. Free text must NEVER be sent to the API for "
|
|
"reformatting under this fix."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_does_not_pass_through_compound_shortened_references():
|
|
"""Chunk 5 task 1: long compound shortened references (multiple
|
|
citations separated by semicolons) must NOT be treated as substantive
|
|
prose, even when they exceed the length threshold and have no
|
|
citation skeleton markers. Surfaced by the HML draft, where ~17 of
|
|
72 notes were compound references like:
|
|
|
|
See Meadowbrooke et al., "Information Behavior and HIV Testing;"
|
|
Kim and Syn, "Health Information on Facebook;" ...
|
|
|
|
These have no parenthesized year, no URL, no vol/no/pp markers, and
|
|
they're 200-400 chars long — so they matched the chunk 3 fix #4
|
|
substantive prose detector and were passed through verbatim, leaving
|
|
curly quotes and incorrect semicolon-quote ordering unfixed.
|
|
|
|
The discriminator: substantive prose doesn't typically use
|
|
semicolons as item separators. Compound references do. So adding
|
|
"no semicolons" as a third condition (alongside "long" and "no
|
|
skeleton markers") cleanly separates the two classes.
|
|
"""
|
|
calls: list[str] = []
|
|
|
|
def recording_caller(system: str, user: str) -> str:
|
|
calls.append(user)
|
|
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
|
|
|
|
# All of these are real-shaped HML compound references (synthesized to
|
|
# match observed patterns). Each is over the 150-char threshold and has
|
|
# no citation skeleton markers (no parenthesized year, no URL, no
|
|
# vol/no/pp). They MUST be sent to the LLM for formatting.
|
|
compound_refs = [
|
|
# 2-reference compound (smallest)
|
|
'Knapp, "Creating Safe and Inclusive Spacing;" Padrón, "Defending Queer Youth;" Pinsky and Brenner, "Threats to Accessing Sexuality Information."',
|
|
# 4-reference compound with "See" prefix
|
|
'See Dilevko and Gottlieb, "Pornography and Bibliographic Access;" Drabinski, "Queering the Catalog;" Billey et al., "Critique of RDA 9.7;" Baucom, "Archival Descriptions of LGBTQ Materials."',
|
|
# 3-reference compound, no prefix
|
|
'Ajayi and Omotayo, "Challenges of HIV/AIDS;" Kachota and Kassim, "Sexual and Reproductive Health Information-Seeking Behaviour;" Bankole and Busayo, "Reproductive Health Information Seeking Behaviour."',
|
|
]
|
|
for ref in compound_refs:
|
|
format_note_entry(ref, caller=recording_caller)
|
|
|
|
assert len(calls) == len(compound_refs), (
|
|
f"some compound references were short-circuited as substantive prose; "
|
|
f"expected {len(compound_refs)} calls, got {len(calls)}. The "
|
|
f"detector is over-matching."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_still_passes_through_genuine_substantive_prose():
|
|
"""Regression test for chunk 5 task 1: tightening the detector with a
|
|
semicolon check must NOT break the genuine substantive-prose pass-
|
|
through that chunk 3 fix #4 was designed for. The Anti-Communist
|
|
notes [8], [9], and [75] (long discursive commentary, NO semicolons)
|
|
must still be preserved verbatim.
|
|
"""
|
|
call_count = 0
|
|
|
|
def counting_caller(system: str, user: str) -> str:
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return "should not be called"
|
|
|
|
# Synthesized to match Anti-Communist note shapes — long discursive
|
|
# prose, no semicolons, no skeleton markers.
|
|
note_8_style = (
|
|
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
|
|
"Morehouse's theory of the talented tenth is celebrated, missing "
|
|
"the note that he himself retracted such a theory later in his "
|
|
"life, as written about by Joy James. The violence of Jim Crow "
|
|
"becomes diminished into many monstrosities, since traditionally "
|
|
"black colleges and universities became monuments to human hope "
|
|
"through philanthropic funding."
|
|
)
|
|
note_75_style = (
|
|
"This letter appeared in multiple archives, including California "
|
|
"State Archives, Berkeley and UCLA special collections, and is "
|
|
"uploaded on UC Santa Barbara Servers."
|
|
)
|
|
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
|
|
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
|
|
assert call_count == 0, (
|
|
"genuine substantive prose was incorrectly sent to the API; the "
|
|
"semicolon-tightening should not have affected these (no semicolons)."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_does_not_pass_through_real_citations():
|
|
"""The substantive-prose guard must NOT match real citations that
|
|
happen to be long. The detection requires absence of citation
|
|
skeleton markers; real citations have at least one (year, URL,
|
|
vol/no/pp, DOI, terminal page). This test pins the negative cases
|
|
so the guard doesn't drift toward over-aggression.
|
|
"""
|
|
calls: list[str] = []
|
|
|
|
def recording_caller(system: str, user: str) -> str:
|
|
calls.append(user)
|
|
# Return validator-passing output so the retry loop doesn't fire
|
|
# and inflate the call count.
|
|
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
|
|
|
|
# Real first-occurrence book note (158 chars, ends with year+period)
|
|
note_6_style = (
|
|
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
|
|
"McCarthyism, Communism, and the Myth of Academic Freedom. "
|
|
"New York: NYU Press, 2023."
|
|
)
|
|
format_note_entry(note_6_style, caller=recording_caller)
|
|
|
|
# Real journal article note (URL marker)
|
|
note_44_style = (
|
|
"Lawrence W.S. Auld, \"The King Report: New Directions in Library "
|
|
"and Information Science Education.\" ACRL College & Research "
|
|
"Libraries News 48, no. 4, 1987. https://crln.acrl.org/example."
|
|
)
|
|
format_note_entry(note_44_style, caller=recording_caller)
|
|
|
|
# Real govt report (parenthesized year + URL)
|
|
note_54_style = (
|
|
"Robert D. Leigh, The California Librarian Education Survey: a "
|
|
"Report to President Robert G. Sproul of the University of "
|
|
"California. (New York: Columbia University, August 1952). "
|
|
"https://hdl.handle.net/example"
|
|
)
|
|
format_note_entry(note_54_style, caller=recording_caller)
|
|
|
|
# Real master's thesis (year at end)
|
|
note_107_style = (
|
|
"Christine Michele Curley, \"The School of the Library,\" "
|
|
"Unpublished Master Thesis, University of California Los "
|
|
"Angeles, 2017."
|
|
)
|
|
format_note_entry(note_107_style, caller=recording_caller)
|
|
|
|
assert len(calls) == 4, (
|
|
"Some real first-occurrence notes were short-circuited as "
|
|
"substantive prose — the detection is too aggressive. Each of "
|
|
"these has at least one citation skeleton marker and must be "
|
|
"sent to the API for reformatting."
|
|
)
|
|
|
|
|
|
def test_format_note_entry_returns_empty_input_verbatim_without_calling_api():
|
|
"""Empty or whitespace-only input must short-circuit the API call and
|
|
return the input verbatim. Without this guard, GPT-5 occasionally breaks
|
|
character on empty input and returns a conversational meta-reply asking
|
|
for content (e.g., "Please paste the citation entry you'd like
|
|
formatted..."). Surfaced by note [66] in the Anti-Communist Formations
|
|
of LIS draft.
|
|
|
|
The whitespace must be preserved verbatim so reassembly in
|
|
cmos.cli.reformat_notes is byte-exact on empty notes — the source's
|
|
spacing is part of the input contract, not noise to be normalized.
|
|
|
|
Critically, the API must NOT be called for empty input — the counting
|
|
caller asserts call_count == 0 to lock in the cost guarantee.
|
|
"""
|
|
call_count = 0
|
|
|
|
def counting_caller(system: str, user: str) -> str:
|
|
nonlocal call_count
|
|
call_count += 1
|
|
return "should not be called"
|
|
|
|
# Each whitespace flavor must round-trip verbatim and not call the API.
|
|
assert format_note_entry("", caller=counting_caller) == ""
|
|
assert format_note_entry(" ", caller=counting_caller) == " "
|
|
assert format_note_entry("\n", caller=counting_caller) == "\n"
|
|
assert format_note_entry("\t \n", caller=counting_caller) == "\t \n"
|
|
|
|
assert call_count == 0, (
|
|
"API caller was invoked on empty input — the short-circuit "
|
|
"guard is missing or not effective. Empty input must NEVER "
|
|
"reach the API."
|
|
)
|