main
3
Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Mark Eaton
|
553240c153 |
v2 chunk 3: bug fixes from real-draft triage + polish
Six independent fixes addressing the Anti-Communist Formations of LIS real-draft test findings (chunk 2c). Four are bug fixes for issues that destroyed user data or wasted API calls; two are polish quality improvements revisited from the deferred list. src/cmos/note_formatter.py — empty input guard (fix #1) Empty / whitespace-only input now short-circuits the API entirely and returns the input verbatim. Surfaced by note [66] in the Anti-Communist draft, where GPT-5 broke character on empty input and returned a conversational meta-reply ("Please paste the citation entry...") that then got substituted into the document. Whitespace preserved so cli.reformat_notes is byte-exact on empty notes. v2 only. src/cmos/note_formatter.py — deprecated Latin guard (fix #2) New Python guard catches the full CMOS-18-deprecated Latin citation set (ibid, idem, id., op. cit., loc. cit.) and returns input verbatim before the API call. Rule 10 in the prompt also rewritten: explicitly says "return verbatim" instead of the old "return cleanest possible full-form note", which the model interpreted as "return empty when there's no information", silently destroying the marker. Surfaced by note [61] = "Ibid.". v2 only. src/cmos/runtime_validator.py — shortened-form carve-out (fix #3) The "must contain italics" check is now skipped when the candidate looks like a CMOS shortened-form note (author last name, optional page, no italic content). Surfaced by ~30 of 107 notes in the Anti-Communist draft that were correctly returned as shortened forms ("Rosen, 7.", "Mitchell, 197.") but rejected by the validator's strict italic check, firing the full retry budget on valid output (~60 wasted API calls per run). The carve-out is conservative: name-token regex + terminal period; doesn't match unstructured prose. Shared infrastructure — affects v1 and v2, no-op in v1's normal workflow because v1 outputs always have italics. src/cmos/note_formatter.py — substantive prose pass-through (fix #4) Long discursive prose with no citation skeleton markers now short-circuits the API and returns the input verbatim. Surfaced by notes [8] (689 chars), [9] (893 chars), [75] (163 chars) in the Anti-Communist draft — substantive notes that the formatter was extracting a single citation from and silently discarding the surrounding commentary. CMOS 14.39 explicitly allows substantive notes; preserving them is the user's explicit policy ("preserve all free text as long as that does not break other formatting"). Detection: length > 150 chars AND no citation skeleton markers (parenthesized year, URL, vol./no./pp., DOI, terminal page or terminal year). Conservative — does not false-positive on legitimate first-occurrence notes. v2 only. src/cmos/formatter.py + note_formatter.py — month/season preservation (polish #5) Both v1 rule 10 (journal article format) and v2 rule 5 (journal article note form) now explicitly instruct the model to preserve (Month YEAR) and (Season YEAR) parentheticals when the source provides them. Surfaced by entries in both Reading_Disrepair and Anti-Communist where (Spring, 1993) and August 1952 were silently collapsed to (1993) and 1952. Both forms are valid CMOS but month/season is more informative when the source has it. Affects v1 and v2 in mirror. src/cmos/note_formatter.py — government documents rule 19 (polish #7) v2 now has an explicit rule mirroring v1's rule 27: government bodies, institutional reports, and similar standalone documents get italicized titles and book-form treatment, NOT quoted-article treatment. Includes the Anti-Communist Senate of California Tenth Report and a hypothetical GAO example. Implicit handling worked in chunk 2c, but explicit rule provides regression protection. v2 only. Tests: +14 net new (across test_note_formatter.py, test_formatter.py, test_runtime_validator.py). Total suite 150/150 (was 136 before this chunk). All v1 tests still passing. Path B status: formatter.py and runtime_validator.py edits cross the v1/v2 boundary, but only by user-explicit approval per the relevant fix discussions. The shared validator was always shared; the v1 formatter rule 10 mirror is a small additive edit that doesn't affect the v1 prompt's existing behavior on its existing inputs. |
||
|
cmos dev
|
8b14dbc9ca |
formatter: linter-guided retry to suppress GPT-5 reasoning drift
GPT-5 is a reasoning model and is not bit-deterministic even at the API level (no temperature override allowed). Variance testing in iter 7 showed ~1 in 6 loop runs hit a nondeterministic regression: in one run the magazine_mead exemplar lost its italic markers around "New Yorker" even though the exact same input had produced a clean output 5 times prior. The scoring linter caught it (rule_magazine_name_italicized) but the formatter still emitted the bad output to the user. This commit adds a runtime guardrail: - New module src/cmos/runtime_validator.py — type-independent structural sanity checks. Operates on a single candidate string with no Exemplar context, because at runtime we don't know the source type. Checks: ends with period, no Ibid., has at least one italic span (almost every CMOS bibliography entry italicizes something), balanced * and " markers. Deliberately weaker than the scoring linter; it's a fast guardrail, not a full validator. - format_bibliography_entry now retries up to DEFAULT_MAX_RETRIES (2) times when the validator rejects a candidate. Independent re-calls are usually enough because the failures are stochastic. If every attempt fails, the LAST attempt is returned (no exception) — the caller still gets something usable, and the failure surfaces through the scoring linter or human review. The retry path costs zero on the common case (1 call per entry); ~1-2% extra calls on noisy drafts. Empirical: 3 consecutive loop runs after this change are scalar 1.000 canary 1.000 (vs 5/6 clean in the variance test before). Sample is too small to claim full suppression but the signal is positive. Also adds a new exemplar journal_multiauthor_secondary_first_last.toml captured from the user's HML draft (Hasanah et al., IJIDI 2024). It exercises the case where a multi-author entry has the first author inverted and the rest in First Last form — which the iter 7 HML run got wrong on one entry. Variance testing showed the exemplar passes 6/6 in isolation, so the original HML failure was nondeterminism, not a missing rule. Keeping the exemplar regardless: it adds canary coverage of a real-world multi-author pattern, no-DOI / JSTOR-URL variant, and the year-suffix author-date holdover stripping. Tests: 85 → 95 (8 new for runtime_validator + 2 new for formatter retry). All passing. |
||
|
cmos dev
|
4cad38ef30 |
Scaffold CMOS 18 reformatter: harness, linter, parser, formatter, CLI
Initial phase-1 baseline of the karpathy/autoresearch-style loop. The formatter module is the inner-loop artifact; parser and linter are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that will force a re-baseline on any rule change. Components: - harness/diff.py: case-sensitive field-level substring diff - harness/score.py: three-axis scoring (field, linter, canary exact) - src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited - src/cmos/parser.py: locate ## Bibliography section, split entries - src/cmos/formatter.py: prompt + OpenAI call with caller injection - src/cmos/cli.py: cmos format path/to/draft.md - scripts/run_loop.py: loop runner with --fake mode for no-API runs - exemplars/: 3 canary seed exemplars (book, journal w/DOI, web), sourced from chicagomanualofstyle.org quick guide Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed exemplars (identity caller fails the linter on every rule). This is the floor the real GPT-5 formatter needs to improve from. |