ocertat/cmos
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

v2 chunk 5: 5 fixes from cross-draft triage (HML + Reading_Disrepair)

Browse Source 
After chunk 4 verification on the Anti-Communist draft, ran v2
against the previously-untested ## Notes sections of HML (72 notes)
and Reading_Disrepair (25 notes) — 97 new notes for 204 total
across 3 drafts. The wider corpus surfaced 5 distinct issues that
weren't visible in the Anti-Communist data alone.

src/cmos/note_formatter.py:

1. Tighten substantive prose detector — exclude semicolon-bearing
   inputs (chunk 5 task 1). The HML draft was dominated by compound
   shortened references like:
     See Author1, "Title;" Author2, "Title;" Author3, "Title."
   17 of 72 HML notes matched this pattern, exceeded the 150-char
   threshold, and had no citation skeleton markers, so they were
   passed through verbatim by the chunk 3 detector. Compound
   references use semicolons as item separators; substantive prose
   doesn't. Adding `if ";" in stripped: return False` cleanly
   separates the two without affecting the genuine substantive
   notes (which contain none).

2. Rule 3 — distinguish comma-inside vs period-inside for
   article/chapter title closing quote based on whether MORE
   content follows the title. Surfaced by 8+ entries across drafts
   producing the wrong `"Title,".` pattern (comma inside followed
   by stray period outside) when the title is the last element of
   the entry. Rule 3 now explicitly says: comma inside when more
   content follows; period inside when title is the last element.

3. New rule 20 — preserve signal phrases verbatim. Surfaced by HML
   [69] and [70] where "For example," was stripped while "See" was
   preserved elsewhere. CMOS notes commonly open with signal
   phrases (See, See also, For example, Cf., Compare, But see,
   Contra, Accord, Quoted in) that indicate how a citation relates
   to the surrounding argument. Stripping them is data loss.

4. Rule 1 — explicit "no comma between author name and 'et al.'"
   in non-inverted note form. Surfaced by HML [69] where "Bignoli
   et al." became "Bignoli, et al." (extra comma). The comma
   before "et al." is a feature of inverted bibliography form,
   not non-inverted note form. Rule 1 now includes WRONG examples.

5. Citation skeleton URL marker — also accept bare-domain URLs
   without the https:// scheme. Surfaced by Reading_Disrepair [1]
   GAO entry which had `files.gao.gov/...` without `https://`,
   so the existing https?:// marker didn't match and the 190-char
   citation was passed through as substantive prose. New pattern:
   `\b\w{2,}(?:\.\w{2,})+/\S*` catches bare domains like
   files.gao.gov/path without false-positiving on common things
   like "e.g./" (single-char tokens excluded by {2,}).

Tests: 157/157 (was 153, +4 net new for prompt-content and regex
verification). The +4 is: compound-reference detection, genuine
substantive prose regression check, rule 3 / rule 20 / rule 1
prompt-content tests, bare-domain URL detection.

Path B: rule 18 (publishers, chunk 4) and rule 20 (signal phrases,
this commit) are v2-only. Rule 1 et al. clarification could in
principle apply to v1 too but v1 doesn't see "et al." in the same
non-inverted form. v1 untouched. 96/96 v1 tests still passing.

Real-draft validation pending — verification re-run against all
3 drafts will follow in the next step.
This commit is contained in:
Mark Eaton
2026-04-11 22:10:30 -04:00
parent 83339e313c
commit c21ca7b58e
2 changed files with 307 additions and 31 deletions
Download Patch File Download Diff File Expand all files Collapse all files
tests/test_note_formatter.py
+188
View File
@@ -112,6 +112,108 @@ def test_system_prompt_mentions_month_or_season_preservation():
assert "preserve" in lower
def test_system_prompt_rule_3_distinguishes_comma_and_period_at_title_end():
"""Chunk 5 task 2: rule 3's punctuation guidance for note form must
distinguish between (a) title followed by more content (comma inside
quote) and (b) title as the last element of the entry (period inside
quote). Surfaced by 8+ entries across drafts producing the wrong
`"Title,".` pattern instead of `"Title."`.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
# Either explicit "last element" wording or both forms shown
assert "last element" in lower or "nothing follows" in lower
# Both punctuation patterns should be mentioned
assert "period inside" in lower or 'title."' in SYSTEM_PROMPT_NOTES.lower()
def test_system_prompt_mentions_signal_phrases_to_preserve():
"""Chunk 5 task 3: explicit rule that signal phrases at the start
of notes must be preserved verbatim. Surfaced by HML [69] and [70]
where "For example," was stripped while "See" was preserved
elsewhere — inconsistent behavior that loses meaningful authorial
intent.
"""
lower = SYSTEM_PROMPT_NOTES.lower()
assert "signal phrase" in lower
# The list should mention at least the common ones that were
# actually observed in the drafts
assert "see also" in lower
assert "for example" in lower
assert "cf." in lower or "compare" in lower
def test_system_prompt_rule_1_says_no_comma_before_et_al():
"""Chunk 5 task 4: rule 1 must explicitly say "et al." follows the
author name with NO comma in note form (non-inverted). Surfaced by
HML [69] where "Bignoli et al." became "Bignoli, et al." with a
spurious comma. CMOS 14.76: in non-inverted note form, the comma
that would precede "et al." in inverted bibliography form is NOT
used.
Test logic: the "no comma" prohibition must appear NEAR the
"et al." mention in rule 1 (within 300 chars), not just somewhere
else in the prompt (the polish #5 month/season rule also says "no
comma" but that's a different context).
"""
text = SYSTEM_PROMPT_NOTES.lower()
# Find all occurrences of "et al" and check if "no comma" is within
# 300 chars before or after each one
found_proximity = False
idx = text.find("et al")
while idx != -1:
window = text[max(0, idx - 300) : min(len(text), idx + 300)]
if "no comma" in window:
found_proximity = True
break
idx = text.find("et al", idx + 1)
# Also accept an explicit wrong-form example
has_wrong_example = (
'"Bignoli, et al."' in SYSTEM_PROMPT_NOTES
or '"Smith, et al."' in SYSTEM_PROMPT_NOTES
or 'wrong: "' in text and "et al" in text
)
assert found_proximity or has_wrong_example, (
"rule 1 must explicitly forbid the comma between author name "
"and 'et al.' in non-inverted note form. The 'no comma' "
"prohibition must appear near the 'et al.' rule, or an "
"explicit wrong-form example must be present."
)
def test_substantive_prose_detector_recognizes_domain_only_urls():
"""Chunk 5 task 5: the citation skeleton marker for URLs must also
recognize bare-domain URLs (without the https:// scheme), so that
real first-occurrence citations with bare-domain URLs aren't
false-positively passed through as substantive prose. Surfaced by
Reading_Disrepair [1] GAO entry which had files.gao.gov/... instead
of https://files.gao.gov/...
"""
# 190+ char input with no parens-year, no vol/no/pp, no DOI marker,
# no terminal page reference — but WITH a bare-domain URL.
input_with_bare_domain = (
"Government Accountability Office. Public Libraries: Many Buildings "
"Are Reported to Be in Poor Condition. GAO-26-107262. 18 December "
"2025. files.gao.gov/reports/GAO-26-107262/index.html"
)
assert len(input_with_bare_domain) > 150 # confirms it would otherwise hit the threshold
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
format_note_entry(input_with_bare_domain, caller=counting_caller)
assert call_count == 1, (
"Citation with bare-domain URL was incorrectly passed through "
"as substantive prose; the URL marker should also recognize "
"bare-domain URLs without the https:// scheme."
)
def test_system_prompt_publisher_exception_list_includes_nyu_press():
"""Chunk 4 task 2: rule 18's publisher exception list should include
NYU Press alongside the existing MIT Press / ALA Editions / MLA.
@@ -366,6 +468,92 @@ def test_format_note_entry_passes_through_substantive_prose_verbatim():
)
def test_format_note_entry_does_not_pass_through_compound_shortened_references():
"""Chunk 5 task 1: long compound shortened references (multiple
citations separated by semicolons) must NOT be treated as substantive
prose, even when they exceed the length threshold and have no
citation skeleton markers. Surfaced by the HML draft, where ~17 of
72 notes were compound references like:
See Meadowbrooke et al., "Information Behavior and HIV Testing;"
Kim and Syn, "Health Information on Facebook;" ...
These have no parenthesized year, no URL, no vol/no/pp markers, and
they're 200-400 chars long — so they matched the chunk 3 fix #4
substantive prose detector and were passed through verbatim, leaving
curly quotes and incorrect semicolon-quote ordering unfixed.
The discriminator: substantive prose doesn't typically use
semicolons as item separators. Compound references do. So adding
"no semicolons" as a third condition (alongside "long" and "no
skeleton markers") cleanly separates the two classes.
"""
calls: list[str] = []
def recording_caller(system: str, user: str) -> str:
calls.append(user)
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
# All of these are real-shaped HML compound references (synthesized to
# match observed patterns). Each is over the 150-char threshold and has
# no citation skeleton markers (no parenthesized year, no URL, no
# vol/no/pp). They MUST be sent to the LLM for formatting.
compound_refs = [
# 2-reference compound (smallest)
'Knapp, "Creating Safe and Inclusive Spacing;" Padrón, "Defending Queer Youth;" Pinsky and Brenner, "Threats to Accessing Sexuality Information."',
# 4-reference compound with "See" prefix
'See Dilevko and Gottlieb, "Pornography and Bibliographic Access;" Drabinski, "Queering the Catalog;" Billey et al., "Critique of RDA 9.7;" Baucom, "Archival Descriptions of LGBTQ Materials."',
# 3-reference compound, no prefix
'Ajayi and Omotayo, "Challenges of HIV/AIDS;" Kachota and Kassim, "Sexual and Reproductive Health Information-Seeking Behaviour;" Bankole and Busayo, "Reproductive Health Information Seeking Behaviour."',
]
for ref in compound_refs:
format_note_entry(ref, caller=recording_caller)
assert len(calls) == len(compound_refs), (
f"some compound references were short-circuited as substantive prose; "
f"expected {len(compound_refs)} calls, got {len(calls)}. The "
f"detector is over-matching."
)
def test_format_note_entry_still_passes_through_genuine_substantive_prose():
"""Regression test for chunk 5 task 1: tightening the detector with a
semicolon check must NOT break the genuine substantive-prose pass-
through that chunk 3 fix #4 was designed for. The Anti-Communist
notes [8], [9], and [75] (long discursive commentary, NO semicolons)
must still be preserved verbatim.
"""
call_count = 0
def counting_caller(system: str, user: str) -> str:
nonlocal call_count
call_count += 1
return "should not be called"
# Synthesized to match Anti-Communist note shapes — long discursive
# prose, no semicolons, no skeleton markers.
note_8_style = (
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
"Morehouse's theory of the talented tenth is celebrated, missing "
"the note that he himself retracted such a theory later in his "
"life, as written about by Joy James. The violence of Jim Crow "
"becomes diminished into many monstrosities, since traditionally "
"black colleges and universities became monuments to human hope "
"through philanthropic funding."
)
note_75_style = (
"This letter appeared in multiple archives, including California "
"State Archives, Berkeley and UCLA special collections, and is "
"uploaded on UC Santa Barbara Servers."
)
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
assert call_count == 0, (
"genuine substantive prose was incorrectly sent to the API; the "
"semicolon-tightening should not have affected these (no semicolons)."
)
def test_format_note_entry_does_not_pass_through_real_citations():
"""The substantive-prose guard must NOT match real citations that
happen to be long. The detection requires absence of citation