Commit Graph
6 Commits
Author SHA1 Message Date
Mark Eaton cad936f576 parser: support diverse md inputs (bold headings, Works Cited, blockquote notes, junk filtering)
- Bibliography heading now accepts bold markers (## **BIBLIOGRAPHY**),
  alternative names (Works Cited, References), and numbered sub-headings
  within the section.
- Original heading text preserved in output instead of hardcoded
  "## Bibliography".
- New find_blockquote_notes() parser for PDF-to-markdown footnote format
  (> N text), wired into CLI as third fallback after pandoc and numbered.
- PDF junk filtered from bibliography entries: bare page numbers,
  blockquote footnotes, download banners, CC license URLs, and short
  running headers.
2026-04-12 00:11:52 -04:00
Mark Eaton 45f51c6341 v2 chunk 2c: numbered-note format support + real-draft test
Extend the v2 parser and CLI to handle [N] text footnote definitions
in addition to pandoc-style [^marker]: text. The numbered format is
what docx-to-text conversion of footnoted Word documents produces.
The user's "Anti-Communist Formations of LIS" draft uses this format
for all 107 of its notes; without this support, v2 was structurally
incapable of running on real-world docx-derived inputs.

src/cmos/parser.py:
- Add NoteDefinition.original_prefix field so reassembly can
  round-trip the source's marker syntax (pandoc input → pandoc
  output, numbered input → numbered output) without consumers
  needing to know which format was matched.
- Update find_notes() to populate original_prefix as "[^N]: ".
- Add find_numbered_notes() targeting "[N] text" definitions.
  Marker must be all digits (rejects [Smith 2020], [foo], etc.);
  caret prefix is rejected (rejects pandoc-style cleanly).

src/cmos/cli.py:
- reformat_notes now auto-detects format: tries find_notes first,
  falls back to find_numbered_notes if no pandoc definitions found.
  Uses definition.original_prefix for reassembly so both formats
  round-trip correctly.

tests/test_parser.py:
- 11 new tests for find_numbered_notes covering: single/multiple
  definitions, multi-digit markers, line number recording, trailing
  whitespace stripping, ignoring pandoc/non-numeric markers, original
  prefix recording, and the actual Anti-Communist draft format.
- 1 new test for find_notes original_prefix population.

tests/test_cli.py:
- 2 new tests for reformat_notes auto-detect: numbered input round-
  trips as numbered output, pandoc input still round-trips as pandoc.

Total suite: 136/136 (was 123, +13 net new). v1 untouched, 96/96
v1 tests still passing.

Real-draft validation: ran cmos format-notes against the 107-note
Anti-Communist Formations of LIS draft (108 calls in parallel via
the existing concurrency=8 thread pool, completed cleanly). Output
saved to /tmp (not committed). All 107 notes preserved through the
pipeline; ~30-40 first-occurrence full notes produced clean CMOS 18
note form; ~30 shortened-form refs correctly left unchanged;
2 real bugs surfaced for the next iteration (empty input → conver-
sational reply, Ibid → empty string), plus several lower-priority
issues (substantive note truncation, retry waste on shortened
forms, lossy month dropping). Not addressed in this chunk per the
"collect signal, don't fix" plan.
2026-04-11 19:16:09 -04:00
Mark Eaton 87cb70b4fb v2 chunk 2b: cli format-notes subcommand and reformat_notes()
Wire the v2 note formatter into the CLI so it can be invoked on
real markdown drafts. The format-notes subcommand mirrors v1's
format subcommand: parser → formatter → reassemble, with concurrent
API calls.

src/cmos/cli.py:
- Import find_notes and format_note_entry alongside the existing
  v1 imports.
- Add reformat_notes(text, formatter, concurrency) that finds
  pandoc-style markdown footnote definitions via find_notes,
  formats each definition's text via the v2 note formatter (or
  an injected fake), and substitutes the formatted text back into
  the original line position. Non-definition lines preserved
  byte-for-byte. Returns text unchanged when no definitions found.
- Register the format-notes argparse subparser with the same
  --concurrency flag as v1's format.
- Dispatch args.command == "format-notes" to reformat_notes.
- Module docstring updated to document both subcommands.

tests/test_cli.py:
- 7 new tests for reformat_notes covering: in-place substitution,
  order preservation under concurrency, prose preservation,
  reference markers staying verbatim, no-op on empty input,
  trailing newline preservation.
- Extended test_python_dash_m_invocation_actually_runs_main to
  also assert "format-notes" appears in --help, catching accidental
  subcommand removal.

Path B integrity: formatter.py, note_formatter.py, parser.py,
linter.py, harness/score.py all unchanged. No LINTER_VERSION
bump. 96/96 v1 tests still passing. Total suite: 123/123.

Real-API end-to-end smoke test on a temp draft with 2 footnote
definitions: both reformatted byte-for-byte, prose and headings
preserved, ## Conclusion section after the notes preserved.
2026-04-11 18:33:45 -04:00
Mark Eaton 22404fbd6b cli: add __main__ guard so python -m cmos.cli actually runs
Without `if __name__ == "__main__": sys.exit(main())` at the bottom
of cli.py, `python -m cmos.cli format <path>` imports the module
but never invokes main(), so the process silently exits 0 with
empty stdout — indistinguishable from a successful run that
produced no output. Discovered during real-draft testing on
2026-04-11.

Adds a regression test that subprocesses the CLI with --help and
asserts on stdout content. argparse --help exits 0 in both broken
and fixed states; stdout content is the only discriminator.

Both invocation paths now work:
- uv run cmos format <path>                (pyproject script entry)
- uv run python -m cmos.cli format <path>  (module invocation)
2026-04-11 17:45:18 -04:00
cmos dev 2bd1bdf0d4 cli: parallelize formatter calls with thread pool
reformat_draft now uses concurrent.futures.ThreadPoolExecutor with a
default of 8 workers. OpenAI SDK calls are synchronous but network-
bound, so threads release the GIL during I/O and give real speedup.
ThreadPoolExecutor.map preserves input order regardless of completion
order, so output is deterministic.

Empirical: HML draft (134 entries) went from ~25 min serial to 1m55s
with concurrency=12. ~12x speedup; further increases hit OpenAI rate
limits.

CLI gains a --concurrency flag (default 8) for tuning per draft size /
rate limit headroom. concurrency=1 forces serial execution for
debugging. New test asserts that order is preserved when concurrent
calls finish out of input order (uses a sleep-by-index fake formatter).
2026-04-11 00:42:56 -04:00
cmos dev 4cad38ef30 Scaffold CMOS 18 reformatter: harness, linter, parser, formatter, CLI
Initial phase-1 baseline of the karpathy/autoresearch-style loop.
The formatter module is the inner-loop artifact; parser and linter
are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that
will force a re-baseline on any rule change.

Components:
- harness/diff.py: case-sensitive field-level substring diff
- harness/score.py: three-axis scoring (field, linter, canary exact)
- src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited
- src/cmos/parser.py: locate ## Bibliography section, split entries
- src/cmos/formatter.py: prompt + OpenAI call with caller injection
- src/cmos/cli.py: cmos format path/to/draft.md
- scripts/run_loop.py: loop runner with --fake mode for no-API runs
- exemplars/: 3 canary seed exemplars (book, journal w/DOI, web),
  sourced from chicagomanualofstyle.org quick guide

Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed
exemplars (identity caller fails the linter on every rule). This is
the floor the real GPT-5 formatter needs to improve from.
2026-04-10 20:48:33 -04:00