Scaffold CMOS 18 reformatter: harness, linter, parser, formatter, CLI
Initial phase-1 baseline of the karpathy/autoresearch-style loop. The formatter module is the inner-loop artifact; parser and linter are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that will force a re-baseline on any rule change. Components: - harness/diff.py: case-sensitive field-level substring diff - harness/score.py: three-axis scoring (field, linter, canary exact) - src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited - src/cmos/parser.py: locate ## Bibliography section, split entries - src/cmos/formatter.py: prompt + OpenAI call with caller injection - src/cmos/cli.py: cmos format path/to/draft.md - scripts/run_loop.py: loop runner with --fake mode for no-API runs - exemplars/: 3 canary seed exemplars (book, journal w/DOI, web), sourced from chicagomanualofstyle.org quick guide Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed exemplars (identity caller fails the linter on every rule). This is the floor the real GPT-5 formatter needs to improve from.
This commit is contained in:
@@ -0,0 +1,45 @@
|
||||
"""Field-level structured diff for CMOS 18 candidate outputs.
|
||||
|
||||
Approach: for each (field, value) in the canonical record, check whether the
|
||||
candidate formatted string contains `str(value)` as a case-sensitive
|
||||
substring. This is deliberately simpler than parsing the candidate — the
|
||||
linter is responsible for structural checks (punctuation, ordering, italics),
|
||||
so the diff only answers: "did the right facts survive the reformat?"
|
||||
|
||||
Frozen harness module. Edits here invalidate prior scores.
|
||||
"""
|
||||
|
||||
from __future__ import annotations
|
||||
|
||||
from dataclasses import dataclass, field
|
||||
|
||||
|
||||
@dataclass
|
||||
class DiffResult:
|
||||
matched: list[str] = field(default_factory=list)
|
||||
missing: list[str] = field(default_factory=list)
|
||||
|
||||
@property
|
||||
def total(self) -> int:
|
||||
return len(self.matched) + len(self.missing)
|
||||
|
||||
@property
|
||||
def rate(self) -> float:
|
||||
if self.total == 0:
|
||||
return 1.0
|
||||
return len(self.matched) / self.total
|
||||
|
||||
|
||||
def field_diff(candidate: str, canonical: dict) -> DiffResult:
|
||||
"""Compare a formatted candidate string against a canonical field dict.
|
||||
|
||||
Each canonical field counts as matched iff its stringified value appears
|
||||
as a case-sensitive substring of `candidate`.
|
||||
"""
|
||||
result = DiffResult()
|
||||
for key, value in canonical.items():
|
||||
if str(value) in candidate:
|
||||
result.matched.append(key)
|
||||
else:
|
||||
result.missing.append(key)
|
||||
return result
|
||||
Reference in New Issue
Block a user