v2 chunk 3: bug fixes from real-draft triage + polish
Six independent fixes addressing the Anti-Communist Formations of LIS real-draft test findings (chunk 2c). Four are bug fixes for issues that destroyed user data or wasted API calls; two are polish quality improvements revisited from the deferred list. src/cmos/note_formatter.py — empty input guard (fix #1) Empty / whitespace-only input now short-circuits the API entirely and returns the input verbatim. Surfaced by note [66] in the Anti-Communist draft, where GPT-5 broke character on empty input and returned a conversational meta-reply ("Please paste the citation entry...") that then got substituted into the document. Whitespace preserved so cli.reformat_notes is byte-exact on empty notes. v2 only. src/cmos/note_formatter.py — deprecated Latin guard (fix #2) New Python guard catches the full CMOS-18-deprecated Latin citation set (ibid, idem, id., op. cit., loc. cit.) and returns input verbatim before the API call. Rule 10 in the prompt also rewritten: explicitly says "return verbatim" instead of the old "return cleanest possible full-form note", which the model interpreted as "return empty when there's no information", silently destroying the marker. Surfaced by note [61] = "Ibid.". v2 only. src/cmos/runtime_validator.py — shortened-form carve-out (fix #3) The "must contain italics" check is now skipped when the candidate looks like a CMOS shortened-form note (author last name, optional page, no italic content). Surfaced by ~30 of 107 notes in the Anti-Communist draft that were correctly returned as shortened forms ("Rosen, 7.", "Mitchell, 197.") but rejected by the validator's strict italic check, firing the full retry budget on valid output (~60 wasted API calls per run). The carve-out is conservative: name-token regex + terminal period; doesn't match unstructured prose. Shared infrastructure — affects v1 and v2, no-op in v1's normal workflow because v1 outputs always have italics. src/cmos/note_formatter.py — substantive prose pass-through (fix #4) Long discursive prose with no citation skeleton markers now short-circuits the API and returns the input verbatim. Surfaced by notes [8] (689 chars), [9] (893 chars), [75] (163 chars) in the Anti-Communist draft — substantive notes that the formatter was extracting a single citation from and silently discarding the surrounding commentary. CMOS 14.39 explicitly allows substantive notes; preserving them is the user's explicit policy ("preserve all free text as long as that does not break other formatting"). Detection: length > 150 chars AND no citation skeleton markers (parenthesized year, URL, vol./no./pp., DOI, terminal page or terminal year). Conservative — does not false-positive on legitimate first-occurrence notes. v2 only. src/cmos/formatter.py + note_formatter.py — month/season preservation (polish #5) Both v1 rule 10 (journal article format) and v2 rule 5 (journal article note form) now explicitly instruct the model to preserve (Month YEAR) and (Season YEAR) parentheticals when the source provides them. Surfaced by entries in both Reading_Disrepair and Anti-Communist where (Spring, 1993) and August 1952 were silently collapsed to (1993) and 1952. Both forms are valid CMOS but month/season is more informative when the source has it. Affects v1 and v2 in mirror. src/cmos/note_formatter.py — government documents rule 19 (polish #7) v2 now has an explicit rule mirroring v1's rule 27: government bodies, institutional reports, and similar standalone documents get italicized titles and book-form treatment, NOT quoted-article treatment. Includes the Anti-Communist Senate of California Tenth Report and a hypothetical GAO example. Implicit handling worked in chunk 2c, but explicit rule provides regression protection. v2 only. Tests: +14 net new (across test_note_formatter.py, test_formatter.py, test_runtime_validator.py). Total suite 150/150 (was 136 before this chunk). All v1 tests still passing. Path B status: formatter.py and runtime_validator.py edits cross the v1/v2 boundary, but only by user-explicit approval per the relevant fix discussions. The shared validator was always shared; the v1 formatter rule 10 mirror is a small additive edit that doesn't affect the v1 prompt's existing behavior on its existing inputs.
This commit is contained in:
Mark Eaton
@@ -79,6 +79,18 @@ Rules you must follow (CMOS 18th edition specifically):
|
||||
10. Journal article format: Lastname, First. "Article Title." *Journal Title*
|
||||
VOLUME, no. ISSUE (YEAR): PAGES. DOI-URL.
|
||||
|
||||
PRESERVE MONTH OR SEASON information when the source provides it.
|
||||
Some scholarly journals organize their issues by month or season
|
||||
rather than (or in addition to) volume/issue numbers, and CMOS
|
||||
accepts both `(YEAR)` and `(Month YEAR)` / `(Season YEAR)` forms
|
||||
in the parenthetical. If the source has a month name (January
|
||||
through December) or a season name (Spring, Summer, Fall, Autumn,
|
||||
Winter), preserve it as `(Month YEAR)` or `(Season YEAR)` — note
|
||||
NO comma between the month/season and the year. Do NOT collapse
|
||||
`(Spring 1993)` to `(1993)` or `August 1952` to `1952` when the
|
||||
source had the month/season; that loses information unnecessarily.
|
||||
If the source has only a year, just use `(YEAR)` as before.
|
||||
|
||||
11. Web page format: Author-or-Organization. "Page Title." Site Name.
|
||||
Effective/Published Date. URL.
|
||||
|
||||
|
||||
+198
-6
@@ -29,12 +29,96 @@ yet implemented.
|
||||
from __future__ import annotations
|
||||
|
||||
import os
|
||||
import re
|
||||
from typing import Callable
|
||||
|
||||
from dotenv import load_dotenv
|
||||
|
||||
from cmos.runtime_validator import validate
|
||||
|
||||
# Pure-token CMOS-18-deprecated Latin citation abbreviations. When the
|
||||
# input matches this exactly (after .strip()), the formatter
|
||||
# short-circuits and returns the input verbatim instead of calling the
|
||||
# API. The model has no information to expand these into a full
|
||||
# citation (shortened-form generation is out of scope), so the only
|
||||
# correct behavior is to preserve the marker for human review.
|
||||
#
|
||||
# Detection is intentionally STRICT — input must match the deprecated
|
||||
# token exactly. Inputs like "Ibid., 47" (token + page number) are
|
||||
# left for the LLM to handle per prompt rule 10, which says return
|
||||
# verbatim in those cases too. This split avoids over-matching real
|
||||
# citations that happen to mention "ibid" inside a longer string.
|
||||
_DEPRECATED_LATIN_RE = re.compile(
|
||||
r"^(?:"
|
||||
r"ibid\.?" # ibid, ibid.
|
||||
r"|idem\.?" # idem, idem.
|
||||
r"|id\." # id. (period required to avoid matching "id" in unrelated contexts)
|
||||
r"|op\.?\s*cit\.?" # op cit, op. cit, op.cit, op cit., etc.
|
||||
r"|loc\.?\s*cit\.?" # loc cit, loc. cit, loc.cit, etc.
|
||||
r")$",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
# Substantive note pass-through (chunk 3 fix #4, Option A). When the
|
||||
# input is long discursive prose with no obvious citation skeleton, the
|
||||
# formatter returns it VERBATIM without calling the API. CMOS 14.39
|
||||
# explicitly allows substantive notes (commentary, qualifications,
|
||||
# cross-references), and the previous behavior extracted whatever
|
||||
# citation it could find and silently discarded the surrounding prose.
|
||||
# Surfaced by notes [8], [9], and [75] in the Anti-Communist Formations
|
||||
# of LIS draft.
|
||||
#
|
||||
# Detection is conservative: input must be longer than the threshold
|
||||
# AND must NOT contain ANY of the citation skeleton markers below.
|
||||
# Real first-occurrence citations virtually always have at least one
|
||||
# skeleton marker (terminal year, terminal page reference, URL, vol.,
|
||||
# no., pp., DOI, parenthesized year). Substantive prose typically has
|
||||
# none of these. The combination of length-floor + marker-absence
|
||||
# catches the egregious cases without false-positiving on legitimate
|
||||
# citations that happen to be long.
|
||||
#
|
||||
# The user's explicit policy (chunk 3 fix #4): preserve ALL free text
|
||||
# as long as that does not break other formatting. Erring on the side
|
||||
# of preservation is intentional.
|
||||
_SUBSTANTIVE_PROSE_LENGTH_THRESHOLD = 150
|
||||
|
||||
_CITATION_SKELETON_MARKERS_RE = re.compile(
|
||||
r"(?:"
|
||||
r"\([^)]*\d{4}[^)]*\)" # parenthesized year — (2024), (NYU Press, 2023), (March 2024)
|
||||
r"|https?://" # URL
|
||||
r"|\bvol\.\s*\d+" # volume number — vol. 7
|
||||
r"|\bno\.\s*\d+" # issue number — no. 4
|
||||
r"|\bpp?\.\s*\d+" # page reference — p. 45, pp. 45-67
|
||||
r"|\bdoi[:.\s]" # DOI marker — doi:, doi., doi space
|
||||
r"|,\s*\d+(?:[-\u2013]\d+)?\.\s*$" # terminal page reference — ", 145." or ", 145-67."
|
||||
r"|\b\d{4}\.?\s*$" # year at end of string — "2023" or "2023."
|
||||
r")",
|
||||
re.IGNORECASE,
|
||||
)
|
||||
|
||||
|
||||
def _looks_like_substantive_prose(text: str) -> bool:
|
||||
"""True if ``text`` looks like substantive note prose, not a citation.
|
||||
|
||||
Conservative detection: only matches input that is BOTH long AND
|
||||
lacks any obvious citation skeleton markers. Designed to catch
|
||||
long discursive notes (689+ chars in the Anti-Communist draft)
|
||||
while preserving legitimate first-occurrence notes — which all
|
||||
have at least one skeleton marker even when long. The shorter
|
||||
[75]-style cases (~163 chars, no markers) are also caught because
|
||||
the threshold is set well below the typical citation length floor
|
||||
when markers are present.
|
||||
|
||||
See ``_SUBSTANTIVE_PROSE_LENGTH_THRESHOLD`` and
|
||||
``_CITATION_SKELETON_MARKERS_RE`` above for the detection details.
|
||||
"""
|
||||
stripped = text.strip()
|
||||
if len(stripped) < _SUBSTANTIVE_PROSE_LENGTH_THRESHOLD:
|
||||
return False
|
||||
if _CITATION_SKELETON_MARKERS_RE.search(stripped):
|
||||
return False
|
||||
return True
|
||||
|
||||
# Default model. Override with OPENAI_MODEL=... in your .env.
|
||||
MODEL = os.environ.get("OPENAI_MODEL", "gpt-5")
|
||||
|
||||
@@ -96,6 +180,19 @@ Rules you must follow (CMOS 18th edition specifically, NOTE form):
|
||||
the quotes, and the colon between (YEAR) and the page is NOT
|
||||
preceded by a space.
|
||||
|
||||
PRESERVE MONTH OR SEASON information when the source provides
|
||||
it. Some scholarly journals organize their issues by month or
|
||||
season rather than (or in addition to) volume/issue numbers, and
|
||||
CMOS accepts both `(YEAR)` and `(Month YEAR)` / `(Season YEAR)`
|
||||
forms in the parenthetical. If the source has a month name
|
||||
(January through December) or a season name (Spring, Summer,
|
||||
Fall, Autumn, Winter), preserve it as `(Month YEAR)` or
|
||||
`(Season YEAR)` — note NO comma between the month/season and
|
||||
the year. Do NOT collapse `(Spring 1993)` to `(1993)` or
|
||||
`August 1952` to `1952` when the source had the month/season;
|
||||
that loses information unnecessarily. If the source has only a
|
||||
year, just use `(YEAR)` as before.
|
||||
|
||||
6. CHAPTER IN EDITED BOOK NOTE FORM:
|
||||
Author, "Chapter Title," in *Book Title*, ed. Editor Names
|
||||
(Publisher, Year), specific-page.
|
||||
@@ -119,12 +216,34 @@ Rules you must follow (CMOS 18th edition specifically, NOTE form):
|
||||
as a full URL: https://doi.org/10.xxxx/yyyy (no "doi:" prefix,
|
||||
no bare DOI).
|
||||
|
||||
10. NO IBID. CMOS 18 deprecates "Ibid." entirely. Do not produce it
|
||||
under any circumstances, even if the input contains it. If the
|
||||
input is an "Ibid." reference, this iteration of the formatter
|
||||
cannot resolve it (shortened-form generation is out of scope).
|
||||
Return the cleanest possible full-form note from whatever
|
||||
information the messy input contains.
|
||||
10. NO DEPRECATED LATIN ABBREVIATIONS. CMOS 18 deprecates the entire
|
||||
family of older Latin citation abbreviations in favor of
|
||||
shortened-form notes:
|
||||
- "ibid." / "ibid" (ibidem — same as previous)
|
||||
- "idem" (the same author as previous)
|
||||
- "id." (also "the same")
|
||||
- "op. cit." (opere citato — in the work cited)
|
||||
- "loc. cit." (loco citato — in the place cited)
|
||||
Do NOT produce ANY of these abbreviations in your output, even
|
||||
if the input contains them.
|
||||
|
||||
If the input contains a deprecated abbreviation that this
|
||||
iteration cannot resolve (because shortened-form generation is
|
||||
out of scope and the formatter has no access to the previously-
|
||||
cited source), return the input VERBATIM. Do NOT return an empty
|
||||
string. Do NOT invent missing context (no fabricated author
|
||||
names, titles, or page numbers). Do NOT silently delete the
|
||||
marker. Returning the input verbatim signals to the user that
|
||||
this entry needs manual resolution against the surrounding notes
|
||||
— losing the marker silently is worse than any other failure
|
||||
mode here.
|
||||
|
||||
The pure-token cases ("Ibid.", "idem", "op. cit.", etc. on their
|
||||
own with no other content) are caught deterministically by a
|
||||
Python-side guard before the API call ever fires; this rule
|
||||
covers the cases where a deprecated abbreviation appears inline
|
||||
with other content (e.g., "Ibid., 47" or "Smith, op. cit., 12").
|
||||
In all such cases: return the input verbatim.
|
||||
|
||||
11. PRESERVE TERMINAL PUNCTUATION inside titles. If a title ends in
|
||||
"?" or "!", keep that punctuation inside the closing quote and
|
||||
@@ -186,6 +305,37 @@ Rules you must follow (CMOS 18th edition specifically, NOTE form):
|
||||
"MLA") stay in that form. When in doubt, prefer the longer
|
||||
form over the abbreviation.
|
||||
|
||||
19. GOVERNMENT DOCUMENTS AND INSTITUTIONAL REPORTS — CMOS 14.272.
|
||||
When the author is a government body, agency, institution, or
|
||||
NGO (e.g., "Government Accountability Office", "U.S. Department
|
||||
of Education", "Senate of the State of California", "World Health
|
||||
Organization", "Pew Research Center", "American Library
|
||||
Association") and the source is a STANDALONE report or document
|
||||
(not an article in a periodical), treat the report like a BOOK:
|
||||
italicize the title with Markdown *...*. Do NOT wrap the title in
|
||||
quotes — that is the article/chapter form, which is wrong for a
|
||||
standalone institutional report.
|
||||
|
||||
Note form:
|
||||
Author-Body, *Title of Report*, Report Number (Publisher, Date),
|
||||
specific-page, URL.
|
||||
|
||||
The Report Number (e.g., "GAO-26-107262") is an identifier and
|
||||
sits after the italicized title with comma separation. The
|
||||
publisher is often the same body as the author (institutional
|
||||
self-publication is common); include both even when they
|
||||
duplicate. If there is no separate publisher and only the author-
|
||||
body is named, that is the publisher.
|
||||
|
||||
Examples:
|
||||
- Government Accountability Office, *Public Libraries: Many
|
||||
Buildings Are Reported to Be in Poor Condition*, GAO-26-107262
|
||||
(Government Accountability Office, December 18, 2025), 4,
|
||||
https://files.gao.gov/reports/GAO-26-107262/index.html.
|
||||
- Senate of the State of California, *Tenth Report: Senate
|
||||
Investigating Committee on Education, the 1951 Hearings*,
|
||||
66, https://hdl.handle.net/2027/mdp.39015070361673.
|
||||
|
||||
Output ONLY the single reformatted note. No preamble, no explanation,
|
||||
no leading number, no code fences.
|
||||
"""
|
||||
@@ -243,6 +393,41 @@ def format_note_entry(
|
||||
Pass a ``caller`` shim to avoid the real API (used by tests and the
|
||||
harness when running with a fake formatter for loop smoke tests).
|
||||
|
||||
Empty input short-circuit: if ``messy_entry`` is empty or contains
|
||||
only whitespace, the function returns it VERBATIM without calling
|
||||
the API. Surfaced by the Anti-Communist Formations of LIS draft
|
||||
(note [66] is an empty placeholder line) — without the guard, GPT-5
|
||||
occasionally breaks character on empty input and returns a
|
||||
conversational meta-reply asking for content. The whitespace is
|
||||
preserved verbatim so reassembly in ``cmos.cli.reformat_notes`` is
|
||||
byte-exact on empty notes; the source's spacing is part of the
|
||||
input contract, not noise to normalize. v2 only — v1's
|
||||
``format_bibliography_entry`` does not currently have this guard.
|
||||
|
||||
Deprecated-Latin short-circuit: if ``messy_entry.strip()`` exactly
|
||||
matches one of the CMOS-18-deprecated Latin citation abbreviations
|
||||
(ibid, idem, id., op. cit., loc. cit.), the function returns the
|
||||
input VERBATIM without calling the API. This iteration cannot
|
||||
resolve shortened-form references (no access to the previously-
|
||||
cited source), and returning empty would silently delete the
|
||||
marker the user needs for manual resolution. Surfaced by note [61]
|
||||
in the Anti-Communist Formations of LIS draft. Detection is
|
||||
strict (token-only); inputs with extra content like "Ibid., 47"
|
||||
fall through to the LLM, which is governed by prompt rule 10.
|
||||
v2 only.
|
||||
|
||||
Substantive prose pass-through: if ``messy_entry`` looks like a
|
||||
substantive note (long discursive prose with no obvious citation
|
||||
skeleton markers), the function returns the input VERBATIM without
|
||||
calling the API. CMOS 14.39 explicitly allows substantive notes,
|
||||
and the previous behavior extracted whatever citation it could
|
||||
find and silently discarded the surrounding prose. Surfaced by
|
||||
notes [8], [9], and [75] in the Anti-Communist Formations of LIS
|
||||
draft. Detection is conservative (length-floor + skeleton-marker
|
||||
absence) — see ``_looks_like_substantive_prose``. The user's
|
||||
explicit policy: preserve ALL free text as long as that does not
|
||||
break other formatting. v2 only.
|
||||
|
||||
Reasoning models (gpt-5, o-series) are nondeterministic. The candidate
|
||||
is checked against ``cmos.runtime_validator`` after each call; if it
|
||||
fails any structural sanity check (missing terminal period, dropped
|
||||
@@ -258,6 +443,13 @@ def format_note_entry(
|
||||
don't raise — the caller still gets something usable, and the
|
||||
failure will surface via the v2 scoring linter or human review).
|
||||
"""
|
||||
stripped = messy_entry.strip()
|
||||
if not stripped:
|
||||
return messy_entry
|
||||
if _DEPRECATED_LATIN_RE.match(stripped):
|
||||
return messy_entry
|
||||
if _looks_like_substantive_prose(messy_entry):
|
||||
return messy_entry
|
||||
call = caller or _openai_caller
|
||||
user_message = build_user_message(messy_entry)
|
||||
last: str = ""
|
||||
|
||||
@@ -1,34 +1,57 @@
|
||||
"""Runtime structural validator — type-independent sanity checks on a single
|
||||
formatted bibliography entry.
|
||||
formatted bibliography or note entry.
|
||||
|
||||
This is the gate used by ``cmos.formatter.format_bibliography_entry`` to
|
||||
decide whether a candidate output is well-formed enough to return, or
|
||||
whether to retry the API call. It runs WITHOUT an Exemplar, because at
|
||||
runtime we have only the messy input and the candidate output — no
|
||||
canonical fields, no source-type metadata.
|
||||
This is the gate used by ``cmos.formatter.format_bibliography_entry`` and
|
||||
``cmos.note_formatter.format_note_entry`` to decide whether a candidate
|
||||
output is well-formed enough to return, or whether to retry the API call.
|
||||
It runs WITHOUT an Exemplar, because at runtime we have only the messy
|
||||
input and the candidate output — no canonical fields, no source-type
|
||||
metadata.
|
||||
|
||||
Because it knows neither the source type nor the canonical fields, it
|
||||
deliberately checks only structural properties that hold for ALL CMOS 18
|
||||
bibliography entries:
|
||||
bibliography entries (and CMOS 18 first-occurrence note entries):
|
||||
|
||||
- Ends with a period.
|
||||
- Contains no "Ibid." (deprecated in CMOS 18).
|
||||
- Contains at least one italic span (``*...*``). This catches one of
|
||||
the most common nondeterministic regressions: GPT-5 occasionally
|
||||
drops italic markers around a journal/book/magazine title. Almost
|
||||
every CMOS bibliography entry italicizes SOMETHING (book title,
|
||||
journal name, magazine name, podcast series, report title), so an
|
||||
entry with no italics at all is highly likely to be malformed.
|
||||
every CMOS full citation italicizes SOMETHING (book title, journal
|
||||
name, magazine name, podcast series, report title), so a full
|
||||
citation with no italics at all is highly likely to be malformed.
|
||||
- Italic markers (``*``) are balanced (even count).
|
||||
- Straight double quotes (``"``) are balanced (even count).
|
||||
|
||||
This is intentionally weaker than ``cmos.linter`` (the scoring linter
|
||||
that needs an Exemplar). The point is to be a fast, deterministic
|
||||
guardrail at runtime, not to validate every CMOS rule.
|
||||
**Shortened-form note carve-out (chunk 3 fix #3, Option D):** the
|
||||
"must contain italics" check is SKIPPED when the candidate looks like
|
||||
a CMOS shortened-form note — author last name (or "Lastname, page")
|
||||
with no title and no italic content. Without this carve-out, the
|
||||
validator rejects valid shortened forms ("Rosen, 7.", "Ettarh.",
|
||||
"Mitchell, 197.") and the formatter's retry loop fires its full budget
|
||||
on correct output. Surfaced by the Anti-Communist Formations of LIS
|
||||
real-draft run where ~30 of 107 notes were valid shortened forms that
|
||||
wasted ~60 API calls between them.
|
||||
|
||||
The carve-out is conservative: it only matches one or two name-token
|
||||
forms (capitalized, allowing apostrophes and hyphens for names like
|
||||
"O'Mara" and "Burden-Stelly") optionally followed by a comma and a
|
||||
page number / page range, ending in a period. Anything more complex
|
||||
(quoted short titles, italicized short titles, longer prose) still
|
||||
falls under the strict "must have italics" check. The goal is to
|
||||
catch the most common shortened-form cases without false positives
|
||||
on actual full-form citations that happen to be short.
|
||||
|
||||
This validator is intentionally weaker than ``cmos.linter`` (the
|
||||
scoring linter that needs an Exemplar). The point is to be a fast,
|
||||
deterministic guardrail at runtime, not to validate every CMOS rule.
|
||||
Shared between v1 (bibliography formatter) and v2 (note formatter)
|
||||
because both use the same retry loop pattern. Edits affect both.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
import re
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@@ -41,6 +64,44 @@ class ValidationResult:
|
||||
return not self.failures
|
||||
|
||||
|
||||
# Shortened-form note pattern. Matches the two most common CMOS shortened
|
||||
# forms that legitimately have no italics:
|
||||
# - Author last name only: "Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."
|
||||
# - Author last name + page: "Rosen, 7.", "Mitchell, 197.", "CBS, 47."
|
||||
# "Lawrence Powell, 45." (multi-word name)
|
||||
# "Mitchell, 137-39." (page range)
|
||||
#
|
||||
# Each name token must start with an uppercase letter (rejects "this is some
|
||||
# prose without italics."). Apostrophes and hyphens are allowed inside name
|
||||
# tokens for names like "O'Mara" and "Burden-Stelly". A multi-word name is
|
||||
# space-separated capitalized tokens. The optional page is a comma followed
|
||||
# by digits, optionally with an ASCII or en-dash range. The whole thing
|
||||
# ends with a required period.
|
||||
#
|
||||
# Does NOT match shortened forms with embedded short titles (quoted or
|
||||
# italicized) — those are harder to disambiguate from real first-occurrence
|
||||
# notes and are left for the strict italics check / future iteration.
|
||||
_SHORTENED_FORM_RE = re.compile(
|
||||
r"^"
|
||||
r"[A-Z][\w'\-]*" # First name token (Lastname, possibly hyphenated/apostrophe)
|
||||
r"(?:\s+[A-Z][\w'\-]*)*" # Optional additional name tokens
|
||||
r"(?:,\s*\d+(?:[-\u2013]\d+)?)?" # Optional ", page" or ", page-range" / page–range
|
||||
r"\.$" # Required terminal period
|
||||
)
|
||||
|
||||
|
||||
def _looks_like_shortened_form(text: str) -> bool:
|
||||
"""True if ``text`` matches the conservative shortened-form pattern.
|
||||
|
||||
Used by ``validate`` to skip the "must have italics" check on
|
||||
legitimate shortened-form notes that have no italic content. Only
|
||||
catches author-only and author-plus-page shapes; more complex
|
||||
shortened forms (with quoted or italicized short titles) are not
|
||||
matched here and remain subject to the strict italic check.
|
||||
"""
|
||||
return _SHORTENED_FORM_RE.match(text.strip()) is not None
|
||||
|
||||
|
||||
def validate(candidate: str) -> ValidationResult:
|
||||
result = ValidationResult()
|
||||
text = candidate.rstrip()
|
||||
@@ -51,9 +112,11 @@ def validate(candidate: str) -> ValidationResult:
|
||||
if "ibid" in text.lower():
|
||||
result.failures.append("contains 'Ibid.' (deprecated in CMOS 18)")
|
||||
|
||||
if "*" not in text:
|
||||
# Strict italic check, with a carve-out for legitimate shortened-form
|
||||
# notes that have no italics by definition. See module docstring.
|
||||
if "*" not in text and not _looks_like_shortened_form(text):
|
||||
result.failures.append(
|
||||
"no italic span found (CMOS bibliography entries usually italicize "
|
||||
"no italic span found (CMOS full citations usually italicize "
|
||||
"a book/journal/magazine/series/report title)"
|
||||
)
|
||||
|
||||
|
||||
@@ -47,6 +47,22 @@ def test_system_prompt_mentions_italic_markers():
|
||||
assert "*" in SYSTEM_PROMPT and "italic" in SYSTEM_PROMPT.lower()
|
||||
|
||||
|
||||
def test_system_prompt_mentions_month_or_season_preservation():
|
||||
"""Polish fix #5: when the source provides month or season information
|
||||
in a scholarly journal date, the formatter should preserve it in CMOS
|
||||
form `(Month YEAR)` or `(Season YEAR)` rather than collapsing to
|
||||
year-only. Surfaced by Reading_Disrepair (v1) and Anti-Communist (v2)
|
||||
real-draft runs where dates like `(Spring, 1993)` and `August 1952`
|
||||
were silently collapsed to `(1993)` and `1952`. Both forms are valid
|
||||
CMOS but month/season is more informative when the source has it.
|
||||
"""
|
||||
lower = SYSTEM_PROMPT.lower()
|
||||
assert "month" in lower
|
||||
assert "season" in lower
|
||||
# And some indication of preservation, not collapse.
|
||||
assert "preserve" in lower
|
||||
|
||||
|
||||
def test_build_user_message_contains_the_messy_input():
|
||||
msg = build_user_message("yu, charles. interior chinatown. 2020")
|
||||
assert "yu, charles. interior chinatown. 2020" in msg
|
||||
|
||||
@@ -53,6 +53,33 @@ def test_system_prompt_mentions_no_ibid():
|
||||
assert "Ibid" in SYSTEM_PROMPT_NOTES or "ibid" in SYSTEM_PROMPT_NOTES.lower()
|
||||
|
||||
|
||||
def test_system_prompt_mentions_broader_deprecated_latin_set():
|
||||
"""Rule 10 was rewritten in fix #2 to cover the broad CMOS 18
|
||||
deprecated set, not just Ibid. The prompt must mention idem,
|
||||
op. cit., loc. cit. so the LLM knows to refuse all of them
|
||||
consistently — the Python guard catches pure-token cases, but
|
||||
the prompt must cover the cases where a deprecated abbreviation
|
||||
appears alongside other content (e.g. "Ibid., 47")."""
|
||||
lower = SYSTEM_PROMPT_NOTES.lower()
|
||||
assert "idem" in lower
|
||||
assert "op. cit" in lower or "op cit" in lower
|
||||
assert "loc. cit" in lower or "loc cit" in lower
|
||||
|
||||
|
||||
def test_system_prompt_rule_10_says_return_verbatim_not_empty():
|
||||
"""Rule 10 must explicitly say "return verbatim" (or equivalent)
|
||||
when the input contains a deprecated abbreviation the formatter
|
||||
cannot resolve. The previous wording said "return the cleanest
|
||||
possible full-form note from whatever information the messy input
|
||||
contains" which the model interpreted as "return empty when there's
|
||||
no information" — destroying the marker the user needs for review.
|
||||
"""
|
||||
lower = SYSTEM_PROMPT_NOTES.lower()
|
||||
assert "verbatim" in lower
|
||||
# And the prompt should explicitly forbid the empty-string output.
|
||||
assert "do not return an empty" in lower or "not return empty" in lower
|
||||
|
||||
|
||||
def test_system_prompt_mentions_publisher_full_form():
|
||||
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
|
||||
preserved 'U of Chicago Press' from the messy input instead of
|
||||
@@ -72,6 +99,35 @@ def test_system_prompt_mentions_publisher_full_form():
|
||||
)
|
||||
|
||||
|
||||
def test_system_prompt_mentions_month_or_season_preservation():
|
||||
"""Polish fix #5 (v2 mirror): preserve month/season info in scholarly
|
||||
journal dates rather than collapsing to year-only. Mirror of the v1
|
||||
test in tests/test_formatter.py — both prompts get the same rule.
|
||||
Surfaced by entries [19] and [54] in the Anti-Communist Formations
|
||||
of LIS draft where (Spring, 1993) and August 1952 were collapsed.
|
||||
"""
|
||||
lower = SYSTEM_PROMPT_NOTES.lower()
|
||||
assert "month" in lower
|
||||
assert "season" in lower
|
||||
assert "preserve" in lower
|
||||
|
||||
|
||||
def test_system_prompt_mentions_government_reports_rule():
|
||||
"""Polish fix #7: v2 should explicitly handle government documents
|
||||
and institutional reports the same way v1's rule 27 does — italicize
|
||||
the title like a book, do NOT wrap in quotes. Implicit handling
|
||||
worked in the chunk 2c real-draft test (entries [11] and [83]) but
|
||||
explicit rule provides regression protection if a future prompt
|
||||
iteration loses this implicit knowledge.
|
||||
"""
|
||||
lower = SYSTEM_PROMPT_NOTES.lower()
|
||||
assert "government" in lower or "institutional" in lower
|
||||
assert "report" in lower
|
||||
# And the key behavior: treat like a book (italicize title), not a
|
||||
# quoted article.
|
||||
assert "italic" in lower # already in the prompt for general italics
|
||||
|
||||
|
||||
def test_system_prompt_mentions_ed_invariant_for_multiple_editors():
|
||||
"""Surfaced by chunk 1 smoke test on chapter_first_doyle: the formatter
|
||||
pluralized 'ed.' to 'eds.' when there were two editors. CMOS NB form
|
||||
@@ -147,3 +203,244 @@ def test_note_formatter_returns_last_attempt_after_max_retries():
|
||||
output = format_note_entry("messy", caller=always_bad, max_retries=2)
|
||||
assert len(attempts) == 3 # 1 initial + 2 retries
|
||||
assert "*" not in output
|
||||
|
||||
|
||||
def test_format_note_entry_short_circuits_deprecated_latin_abbreviations():
|
||||
"""When the input is ONLY a CMOS-18-deprecated Latin citation
|
||||
abbreviation (ibid, idem, id., op. cit., loc. cit.), the formatter
|
||||
must short-circuit the API and return the input verbatim. Surfaced
|
||||
by note [61] in the Anti-Communist Formations of LIS draft, where
|
||||
the formatter was given "Ibid." and (after retrying through the
|
||||
runtime validator) returned an empty string — silently destroying
|
||||
the marker that the user needs to find later for manual resolution.
|
||||
|
||||
The deprecated set per CMOS 18:
|
||||
ibid, ibid., idem, id., op. cit., loc. cit.
|
||||
plus common spelling/spacing variants (Op.Cit., loc cit, etc.).
|
||||
|
||||
The guard MUST be deterministic (Python-side, not LLM-side) and
|
||||
MUST NOT call the API. Cost guarantee enforced via counting caller.
|
||||
The whitespace of the original input is preserved verbatim so
|
||||
reassembly in cmos.cli.reformat_notes is byte-exact.
|
||||
"""
|
||||
call_count = 0
|
||||
|
||||
def counting_caller(system: str, user: str) -> str:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return "should not be called"
|
||||
|
||||
deprecated_inputs = [
|
||||
"Ibid.",
|
||||
"ibid",
|
||||
"IBID.",
|
||||
"Idem",
|
||||
"idem.",
|
||||
"id.",
|
||||
"Id.",
|
||||
"op. cit.",
|
||||
"Op. Cit.",
|
||||
"op.cit.",
|
||||
"Op Cit",
|
||||
"loc. cit.",
|
||||
"Loc. Cit.",
|
||||
"loc.cit.",
|
||||
# With surrounding whitespace preserved verbatim
|
||||
" Ibid. ",
|
||||
"\tIbid.\n",
|
||||
]
|
||||
for inp in deprecated_inputs:
|
||||
result = format_note_entry(inp, caller=counting_caller)
|
||||
assert result == inp, (
|
||||
f"Input {inp!r} must round-trip verbatim; got {result!r}"
|
||||
)
|
||||
|
||||
assert call_count == 0, (
|
||||
f"API caller was invoked {call_count} times for deprecated "
|
||||
"Latin abbreviations — the short-circuit guard is missing or "
|
||||
"not effective. Deprecated tokens must NEVER reach the API."
|
||||
)
|
||||
|
||||
|
||||
def test_format_note_entry_does_not_short_circuit_normal_citations():
|
||||
"""The deprecated-token guard must NOT match real citations that
|
||||
happen to start with one of the tokens. e.g. 'Ibid., 47' is a
|
||||
common usage where a page number is appended; for now we let the
|
||||
LLM see those (the prompt rule covers them). Pure-token-only
|
||||
inputs short-circuit; anything with extra content goes to the API.
|
||||
"""
|
||||
calls: list[str] = []
|
||||
|
||||
def recording_caller(system: str, user: str) -> str:
|
||||
calls.append(user)
|
||||
return "Charles Yu, *Interior Chinatown* (Pantheon Books, 2020), 45."
|
||||
|
||||
# Real-looking inputs that contain "ibid" but are not pure tokens
|
||||
not_short_circuited = [
|
||||
"Ibid., 47.", # ibid + page — common usage
|
||||
"Smith, Ibid., 47.",
|
||||
"Title with the word ibid in it.",
|
||||
"Charles Yu, Interior Chinatown.", # totally unrelated
|
||||
]
|
||||
for inp in not_short_circuited:
|
||||
format_note_entry(inp, caller=recording_caller)
|
||||
|
||||
assert len(calls) == len(not_short_circuited), (
|
||||
"Some non-pure-token inputs were short-circuited — guard is "
|
||||
"too aggressive."
|
||||
)
|
||||
|
||||
|
||||
def test_format_note_entry_passes_through_substantive_prose_verbatim():
|
||||
"""When the input is substantive prose — long discursive commentary
|
||||
with no obvious citation skeleton — the formatter must return it
|
||||
VERBATIM without calling the API. CMOS 14.39 explicitly allows
|
||||
substantive notes (commentary, qualifications, cross-references).
|
||||
The previous behavior extracted whatever citation it could find and
|
||||
silently discarded the surrounding prose, which is data loss.
|
||||
|
||||
Surfaced by notes [8], [9], and [75] in the Anti-Communist
|
||||
Formations of LIS draft. The user's explicit policy (chunk 3
|
||||
fix #4): preserve ALL free text, as long as that does not break
|
||||
other formatting.
|
||||
|
||||
Detection rule (conservative): length > 150 chars AND no obvious
|
||||
citation skeleton markers (parenthesized year, URL, vol./no./pp.,
|
||||
DOI, terminal page reference, terminal year). This catches the
|
||||
egregious cases without false-positiving on legitimate citations,
|
||||
which all have at least one skeleton marker.
|
||||
"""
|
||||
call_count = 0
|
||||
|
||||
def counting_caller(system: str, user: str) -> str:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return "should not be called"
|
||||
|
||||
# [8]-style: long discursive note about DuBois/Morehouse, no markers
|
||||
note_8_style = (
|
||||
"W.E.B. DuBois' popularization of philanthropist Henry Lyman "
|
||||
"Morehouse's theory of the talented tenth is celebrated, missing "
|
||||
"the note that he himself retracted such a theory later in his "
|
||||
"life, as written about by Joy James. The violence of Jim Crow "
|
||||
"becomes diminished into many monstrosities, since traditionally "
|
||||
"black colleges and universities became monuments to human hope "
|
||||
"through philanthropic funding."
|
||||
)
|
||||
assert format_note_entry(note_8_style, caller=counting_caller) == note_8_style
|
||||
|
||||
# [9]-style: discursive about Wilder/Murch, no markers
|
||||
note_9_style = (
|
||||
"Craig Steven Wilder's Ebony and Ivy discusses the political "
|
||||
"economy of anti-Blackness in higher education, and the influence "
|
||||
"of not only capital and capitalists in the creation of "
|
||||
"Historically Black Colleges and Universities and white elite "
|
||||
"universities, but also enslaved labor to support the latter."
|
||||
)
|
||||
assert format_note_entry(note_9_style, caller=counting_caller) == note_9_style
|
||||
|
||||
# [75]-style: shorter cross-archive prose, no markers
|
||||
note_75_style = (
|
||||
"This letter appeared in multiple archives, including California "
|
||||
"State Archives, Berkeley and UCLA special collections, and is "
|
||||
"uploaded on UC Santa Barbara Servers."
|
||||
)
|
||||
assert format_note_entry(note_75_style, caller=counting_caller) == note_75_style
|
||||
|
||||
assert call_count == 0, (
|
||||
f"API caller was invoked {call_count} times for substantive prose — "
|
||||
"the substantive-note pass-through guard is missing or not "
|
||||
"effective. Free text must NEVER be sent to the API for "
|
||||
"reformatting under this fix."
|
||||
)
|
||||
|
||||
|
||||
def test_format_note_entry_does_not_pass_through_real_citations():
|
||||
"""The substantive-prose guard must NOT match real citations that
|
||||
happen to be long. The detection requires absence of citation
|
||||
skeleton markers; real citations have at least one (year, URL,
|
||||
vol/no/pp, DOI, terminal page). This test pins the negative cases
|
||||
so the guard doesn't drift toward over-aggression.
|
||||
"""
|
||||
calls: list[str] = []
|
||||
|
||||
def recording_caller(system: str, user: str) -> str:
|
||||
calls.append(user)
|
||||
# Return validator-passing output so the retry loop doesn't fire
|
||||
# and inflate the call count.
|
||||
return "Fake Author, *Fake Title* (Fake Publisher, 2024), 1."
|
||||
|
||||
# Real first-occurrence book note (158 chars, ends with year+period)
|
||||
note_6_style = (
|
||||
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
|
||||
"McCarthyism, Communism, and the Myth of Academic Freedom. "
|
||||
"New York: NYU Press, 2023."
|
||||
)
|
||||
format_note_entry(note_6_style, caller=recording_caller)
|
||||
|
||||
# Real journal article note (URL marker)
|
||||
note_44_style = (
|
||||
"Lawrence W.S. Auld, \"The King Report: New Directions in Library "
|
||||
"and Information Science Education.\" ACRL College & Research "
|
||||
"Libraries News 48, no. 4, 1987. https://crln.acrl.org/example."
|
||||
)
|
||||
format_note_entry(note_44_style, caller=recording_caller)
|
||||
|
||||
# Real govt report (parenthesized year + URL)
|
||||
note_54_style = (
|
||||
"Robert D. Leigh, The California Librarian Education Survey: a "
|
||||
"Report to President Robert G. Sproul of the University of "
|
||||
"California. (New York: Columbia University, August 1952). "
|
||||
"https://hdl.handle.net/example"
|
||||
)
|
||||
format_note_entry(note_54_style, caller=recording_caller)
|
||||
|
||||
# Real master's thesis (year at end)
|
||||
note_107_style = (
|
||||
"Christine Michele Curley, \"The School of the Library,\" "
|
||||
"Unpublished Master Thesis, University of California Los "
|
||||
"Angeles, 2017."
|
||||
)
|
||||
format_note_entry(note_107_style, caller=recording_caller)
|
||||
|
||||
assert len(calls) == 4, (
|
||||
"Some real first-occurrence notes were short-circuited as "
|
||||
"substantive prose — the detection is too aggressive. Each of "
|
||||
"these has at least one citation skeleton marker and must be "
|
||||
"sent to the API for reformatting."
|
||||
)
|
||||
|
||||
|
||||
def test_format_note_entry_returns_empty_input_verbatim_without_calling_api():
|
||||
"""Empty or whitespace-only input must short-circuit the API call and
|
||||
return the input verbatim. Without this guard, GPT-5 occasionally breaks
|
||||
character on empty input and returns a conversational meta-reply asking
|
||||
for content (e.g., "Please paste the citation entry you'd like
|
||||
formatted..."). Surfaced by note [66] in the Anti-Communist Formations
|
||||
of LIS draft.
|
||||
|
||||
The whitespace must be preserved verbatim so reassembly in
|
||||
cmos.cli.reformat_notes is byte-exact on empty notes — the source's
|
||||
spacing is part of the input contract, not noise to be normalized.
|
||||
|
||||
Critically, the API must NOT be called for empty input — the counting
|
||||
caller asserts call_count == 0 to lock in the cost guarantee.
|
||||
"""
|
||||
call_count = 0
|
||||
|
||||
def counting_caller(system: str, user: str) -> str:
|
||||
nonlocal call_count
|
||||
call_count += 1
|
||||
return "should not be called"
|
||||
|
||||
# Each whitespace flavor must round-trip verbatim and not call the API.
|
||||
assert format_note_entry("", caller=counting_caller) == ""
|
||||
assert format_note_entry(" ", caller=counting_caller) == " "
|
||||
assert format_note_entry("\n", caller=counting_caller) == "\n"
|
||||
assert format_note_entry("\t \n", caller=counting_caller) == "\t \n"
|
||||
|
||||
assert call_count == 0, (
|
||||
"API caller was invoked on empty input — the short-circuit "
|
||||
"guard is missing or not effective. Empty input must NEVER "
|
||||
"reach the API."
|
||||
)
|
||||
|
||||
@@ -50,6 +50,78 @@ def test_missing_italics_anywhere_fails():
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
# --- Shortened-form note carve-out (chunk 3 fix #3, Option D) -------------
|
||||
|
||||
|
||||
def test_shortened_form_author_only_passes_without_italics():
|
||||
"""CMOS shortened note form for an author-only reference (no page,
|
||||
no title) is just the author's last name with a period: 'Ettarh.',
|
||||
'Murch.', 'O'Mara.'. These contain no italics by definition.
|
||||
Without a carve-out, the validator's "must have italics" rule
|
||||
rejects them and the formatter's retry loop fires its full budget
|
||||
on valid output. Surfaced by the Anti-Communist Formations of LIS
|
||||
real-draft run; ~30 of 107 notes were correct shortened-forms
|
||||
that wasted ~60 API calls retrying.
|
||||
|
||||
The carve-out: if a candidate has no italics AND looks like a
|
||||
shortened-form note (short, name-token shape, optional page,
|
||||
terminal period), the missing-italics check is skipped.
|
||||
"""
|
||||
for candidate in ["Ettarh.", "Murch.", "O'Mara.", "Burden-Stelly."]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form author-only note {candidate!r} should pass "
|
||||
f"validation; got failures {result.failures}"
|
||||
)
|
||||
|
||||
|
||||
def test_shortened_form_author_plus_page_passes_without_italics():
|
||||
"""The other common CMOS shortened form is author last name + page:
|
||||
'Rosen, 7.', 'Mitchell, 197.', 'CBS, 47.'. Same carve-out applies."""
|
||||
for candidate in [
|
||||
"Rosen, 7.",
|
||||
"Mitchell, 197.",
|
||||
"CBS, 47.",
|
||||
"Seybold, 282.",
|
||||
"Lawrence Powell, 45.", # multi-word name
|
||||
"Mitchell, 137-39.", # page range with hyphen
|
||||
]:
|
||||
result = validate(candidate)
|
||||
assert result.passed is True, (
|
||||
f"shortened-form note {candidate!r} should pass validation; "
|
||||
f"got failures {result.failures}"
|
||||
)
|
||||
|
||||
|
||||
def test_long_no_italic_output_still_fails():
|
||||
"""The carve-out must NOT be a free pass for any short string.
|
||||
A long output with no italics is still likely a malformed
|
||||
full-form citation (model dropped the italic title), and the
|
||||
retry loop should still fire on it.
|
||||
"""
|
||||
candidate = (
|
||||
"Steve Batterson, The Prosecution of Professor Chandler Davis: "
|
||||
"McCarthyism, Communism, and the Myth of Academic Freedom (NYU Press, 2023)."
|
||||
)
|
||||
result = validate(candidate)
|
||||
assert result.passed is False
|
||||
assert any("italic" in f.lower() for f in result.failures)
|
||||
|
||||
|
||||
def test_short_but_unstructured_no_italic_output_still_fails():
|
||||
"""A short but non-name-shaped string (e.g., a stray sentence
|
||||
fragment with no name structure) should still fail. The carve-out
|
||||
is NOT just "short and ends with period" — it requires the
|
||||
candidate to look like a shortened-form note specifically.
|
||||
"""
|
||||
candidate = "this is some prose without italics."
|
||||
result = validate(candidate)
|
||||
assert result.passed is False, (
|
||||
"lowercase prose fragment must not be accepted as a shortened-form "
|
||||
"note; the carve-out should require name-token shape"
|
||||
)
|
||||
|
||||
|
||||
def test_unbalanced_italic_markers_fails():
|
||||
candidate = "Yu, Charles. *Interior Chinatown. Pantheon Books, 2020."
|
||||
result = validate(candidate)
|
||||
|
||||
Reference in New Issue
Block a user