main
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mark Eaton
|
83339e313c |
v2 chunk 4: revert validator carve-out + expand publisher exceptions
Two follow-ups based on the chunk 3 verification re-run findings. src/cmos/runtime_validator.py — revert chunk 3 fix #3 (carve-out) Chunk 3 fix #3 added a carve-out so shortened-form notes ("Rosen, 7.", "Ettarh.") would pass the validator without italics, saving retry cost. Chunk 3 verification on the Anti-Communist draft showed the carve-out also let GPT-5's variance produce under-italicized variants of shortened forms WITH short titles (e.g., "A Restudy, 73." instead of the more CMOS-correct "*A Restudy*, 73."). The user explicitly chose accuracy over the cost saving and asked for the carve-out to be reverted. The strict italics check now applies uniformly. Simple shortened forms (Mitchell, 197., Ettarh.) get retried unnecessarily and waste API cost without producing better output. Shortened forms with short titles get a fair shot at the italicized version on the retry. Cost ↑, accuracy ↑. Removes _SHORTENED_FORM_RE, _looks_like_shortened_form, and the carve-out check from validate(). Updates module docstring with history note. Inverts the 2 carve-out tests in test_runtime_validator.py to assert shortened forms now FAIL the strict check, documenting the design intent for future reviewers. src/cmos/note_formatter.py — expand rule 18 publisher exception list Chunk 3 verification showed [6] Batterson getting "NYU Press" expanded to "New York University Press", losing the publisher's canonical brand. Rule 18's exception list previously only named MIT Press, ALA Editions, and MLA. Expanded to include NYU Press, Routledge, IEEE Press, ACM Press, WHO Press, plus "Pew Research Center" as a non-Press canonical example. Also added a "when in doubt" guidance paragraph: if the abbreviation contains "Press" and is widely used as the publisher's own branding, leave it alone. The risk of losing a brand name (NYU Press) is worse than the risk of leaving an obscure abbreviation. v2 only. Tests: 151/151 green (was 150). The +1 is the new NYU Press prompt-content test; the 2 inverted runtime_validator tests stayed at the same count. Path B: runtime_validator change crosses the v1/v2 boundary (shared infrastructure), per the same approval that authorized the original chunk 3 fix #3. v1's bibliography formatter is unaffected in practice because v1 outputs always have italics. |
||
|
Mark Eaton
|
553240c153 |
v2 chunk 3: bug fixes from real-draft triage + polish
Six independent fixes addressing the Anti-Communist Formations of LIS real-draft test findings (chunk 2c). Four are bug fixes for issues that destroyed user data or wasted API calls; two are polish quality improvements revisited from the deferred list. src/cmos/note_formatter.py — empty input guard (fix #1) Empty / whitespace-only input now short-circuits the API entirely and returns the input verbatim. Surfaced by note [66] in the Anti-Communist draft, where GPT-5 broke character on empty input and returned a conversational meta-reply ("Please paste the citation entry...") that then got substituted into the document. Whitespace preserved so cli.reformat_notes is byte-exact on empty notes. v2 only. src/cmos/note_formatter.py — deprecated Latin guard (fix #2) New Python guard catches the full CMOS-18-deprecated Latin citation set (ibid, idem, id., op. cit., loc. cit.) and returns input verbatim before the API call. Rule 10 in the prompt also rewritten: explicitly says "return verbatim" instead of the old "return cleanest possible full-form note", which the model interpreted as "return empty when there's no information", silently destroying the marker. Surfaced by note [61] = "Ibid.". v2 only. src/cmos/runtime_validator.py — shortened-form carve-out (fix #3) The "must contain italics" check is now skipped when the candidate looks like a CMOS shortened-form note (author last name, optional page, no italic content). Surfaced by ~30 of 107 notes in the Anti-Communist draft that were correctly returned as shortened forms ("Rosen, 7.", "Mitchell, 197.") but rejected by the validator's strict italic check, firing the full retry budget on valid output (~60 wasted API calls per run). The carve-out is conservative: name-token regex + terminal period; doesn't match unstructured prose. Shared infrastructure — affects v1 and v2, no-op in v1's normal workflow because v1 outputs always have italics. src/cmos/note_formatter.py — substantive prose pass-through (fix #4) Long discursive prose with no citation skeleton markers now short-circuits the API and returns the input verbatim. Surfaced by notes [8] (689 chars), [9] (893 chars), [75] (163 chars) in the Anti-Communist draft — substantive notes that the formatter was extracting a single citation from and silently discarding the surrounding commentary. CMOS 14.39 explicitly allows substantive notes; preserving them is the user's explicit policy ("preserve all free text as long as that does not break other formatting"). Detection: length > 150 chars AND no citation skeleton markers (parenthesized year, URL, vol./no./pp., DOI, terminal page or terminal year). Conservative — does not false-positive on legitimate first-occurrence notes. v2 only. src/cmos/formatter.py + note_formatter.py — month/season preservation (polish #5) Both v1 rule 10 (journal article format) and v2 rule 5 (journal article note form) now explicitly instruct the model to preserve (Month YEAR) and (Season YEAR) parentheticals when the source provides them. Surfaced by entries in both Reading_Disrepair and Anti-Communist where (Spring, 1993) and August 1952 were silently collapsed to (1993) and 1952. Both forms are valid CMOS but month/season is more informative when the source has it. Affects v1 and v2 in mirror. src/cmos/note_formatter.py — government documents rule 19 (polish #7) v2 now has an explicit rule mirroring v1's rule 27: government bodies, institutional reports, and similar standalone documents get italicized titles and book-form treatment, NOT quoted-article treatment. Includes the Anti-Communist Senate of California Tenth Report and a hypothetical GAO example. Implicit handling worked in chunk 2c, but explicit rule provides regression protection. v2 only. Tests: +14 net new (across test_note_formatter.py, test_formatter.py, test_runtime_validator.py). Total suite 150/150 (was 136 before this chunk). All v1 tests still passing. Path B status: formatter.py and runtime_validator.py edits cross the v1/v2 boundary, but only by user-explicit approval per the relevant fix discussions. The shared validator was always shared; the v1 formatter rule 10 mirror is a small additive edit that doesn't affect the v1 prompt's existing behavior on its existing inputs. |
||
|
cmos dev
|
8b14dbc9ca |
formatter: linter-guided retry to suppress GPT-5 reasoning drift
GPT-5 is a reasoning model and is not bit-deterministic even at the API level (no temperature override allowed). Variance testing in iter 7 showed ~1 in 6 loop runs hit a nondeterministic regression: in one run the magazine_mead exemplar lost its italic markers around "New Yorker" even though the exact same input had produced a clean output 5 times prior. The scoring linter caught it (rule_magazine_name_italicized) but the formatter still emitted the bad output to the user. This commit adds a runtime guardrail: - New module src/cmos/runtime_validator.py — type-independent structural sanity checks. Operates on a single candidate string with no Exemplar context, because at runtime we don't know the source type. Checks: ends with period, no Ibid., has at least one italic span (almost every CMOS bibliography entry italicizes something), balanced * and " markers. Deliberately weaker than the scoring linter; it's a fast guardrail, not a full validator. - format_bibliography_entry now retries up to DEFAULT_MAX_RETRIES (2) times when the validator rejects a candidate. Independent re-calls are usually enough because the failures are stochastic. If every attempt fails, the LAST attempt is returned (no exception) — the caller still gets something usable, and the failure surfaces through the scoring linter or human review. The retry path costs zero on the common case (1 call per entry); ~1-2% extra calls on noisy drafts. Empirical: 3 consecutive loop runs after this change are scalar 1.000 canary 1.000 (vs 5/6 clean in the variance test before). Sample is too small to claim full suppression but the signal is positive. Also adds a new exemplar journal_multiauthor_secondary_first_last.toml captured from the user's HML draft (Hasanah et al., IJIDI 2024). It exercises the case where a multi-author entry has the first author inverted and the rest in First Last form — which the iter 7 HML run got wrong on one entry. Variance testing showed the exemplar passes 6/6 in isolation, so the original HML failure was nondeterminism, not a missing rule. Keeping the exemplar regardless: it adds canary coverage of a real-world multi-author pattern, no-DOI / JSTOR-URL variant, and the year-suffix author-date holdover stripping. Tests: 85 → 95 (8 new for runtime_validator + 2 new for formatter retry). All passing. |