v2 chunk 2c: numbered-note format support + real-draft test

Extend the v2 parser and CLI to handle [N] text footnote definitions
in addition to pandoc-style [^marker]: text. The numbered format is
what docx-to-text conversion of footnoted Word documents produces.
The user's "Anti-Communist Formations of LIS" draft uses this format
for all 107 of its notes; without this support, v2 was structurally
incapable of running on real-world docx-derived inputs.

src/cmos/parser.py:
- Add NoteDefinition.original_prefix field so reassembly can
  round-trip the source's marker syntax (pandoc input → pandoc
  output, numbered input → numbered output) without consumers
  needing to know which format was matched.
- Update find_notes() to populate original_prefix as "[^N]: ".
- Add find_numbered_notes() targeting "[N] text" definitions.
  Marker must be all digits (rejects [Smith 2020], [foo], etc.);
  caret prefix is rejected (rejects pandoc-style cleanly).

src/cmos/cli.py:
- reformat_notes now auto-detects format: tries find_notes first,
  falls back to find_numbered_notes if no pandoc definitions found.
  Uses definition.original_prefix for reassembly so both formats
  round-trip correctly.

tests/test_parser.py:
- 11 new tests for find_numbered_notes covering: single/multiple
  definitions, multi-digit markers, line number recording, trailing
  whitespace stripping, ignoring pandoc/non-numeric markers, original
  prefix recording, and the actual Anti-Communist draft format.
- 1 new test for find_notes original_prefix population.

tests/test_cli.py:
- 2 new tests for reformat_notes auto-detect: numbered input round-
  trips as numbered output, pandoc input still round-trips as pandoc.

Total suite: 136/136 (was 123, +13 net new). v1 untouched, 96/96
v1 tests still passing.

Real-draft validation: ran cmos format-notes against the 107-note
Anti-Communist Formations of LIS draft (108 calls in parallel via
the existing concurrency=8 thread pool, completed cleanly). Output
saved to /tmp (not committed). All 107 notes preserved through the
pipeline; ~30-40 first-occurrence full notes produced clean CMOS 18
note form; ~30 shortened-form refs correctly left unchanged;
2 real bugs surfaced for the next iteration (empty input → conver-
sational reply, Ibid → empty string), plus several lower-priority
issues (substantive note truncation, retry waste on shortened
forms, lossy month dropping). Not addressed in this chunk per the
"collect signal, don't fix" plan.
This commit is contained in:
Mark Eaton
2026-04-11 19:16:09 -04:00
parent 87cb70b4fb
commit 45f51c6341
4 changed files with 286 additions and 36 deletions
+31
View File
@@ -195,6 +195,37 @@ def test_reformat_notes_preserves_trailing_newline():
assert not reformat_notes(without_trailing, formatter=_fake_note_formatter).endswith("\n")
def test_reformat_notes_auto_detects_numbered_format():
"""Real drafts converted from docx use [N] text format, not pandoc.
reformat_notes must auto-detect: try pandoc first, fall back to
numbered. Reassembly must use the original prefix (e.g. "[1] ") so
the output round-trips in the source's marker syntax."""
draft = """\
## Notes
[1] yu, charles. interior chinatown. 2020. p 45.
[2] kwon, hyeyoung. inclusion work. 2022, p 1830.
"""
output = reformat_notes(draft, formatter=_fake_note_formatter)
# Numbered prefix preserved, NOT rewritten as pandoc.
assert "[1] NOTE_FORMATTED(yu, charles. interior chinatown. 2020. p 45.)" in output
assert "[2] NOTE_FORMATTED(kwon, hyeyoung. inclusion work. 2022, p 1830.)" in output
# No pandoc-style markers should appear in the output (we did not
# auto-convert numbered to pandoc).
assert "[^1]" not in output
assert "[^2]" not in output
def test_reformat_notes_pandoc_input_still_round_trips_as_pandoc():
"""Regression: after the auto-detect change, pandoc-format input
must still produce pandoc-format output. The original_prefix path
must work for both formats."""
draft = "[^1]: messy.\n"
output = reformat_notes(draft, formatter=_fake_note_formatter)
assert "[^1]: NOTE_FORMATTED(messy.)" in output
assert "[1] " not in output # numbered prefix must NOT appear
def test_reformat_notes_preserves_order_under_concurrency():
"""With concurrent execution the formatter is called on all definitions
in parallel; the output must reassemble them in input order regardless