cmos dev
4cad38ef30
Initial phase-1 baseline of the karpathy/autoresearch-style loop. The formatter module is the inner-loop artifact; parser and linter are infra. The linter carries a LINTER_VERSION hash (v0.2.0) that will force a re-baseline on any rule change. Components: - harness/diff.py: case-sensitive field-level substring diff - harness/score.py: three-axis scoring (field, linter, canary exact) - src/cmos/linter.py: 9 CMOS 18 structural rules, each Purdue/CMOS cited - src/cmos/parser.py: locate ## Bibliography section, split entries - src/cmos/formatter.py: prompt + OpenAI call with caller injection - src/cmos/cli.py: cmos format path/to/draft.md - scripts/run_loop.py: loop runner with --fake mode for no-API runs - exemplars/: 3 canary seed exemplars (book, journal w/DOI, web), sourced from chicagomanualofstyle.org quick guide Tests: 48 passing. Fake-mode baseline scalar = 0.000 on the 3 seed exemplars (identity caller fails the linter on every rule). This is the floor the real GPT-5 formatter needs to improve from.
46 lines
1.4 KiB
Python
46 lines
1.4 KiB
Python
"""Field-level structured diff for CMOS 18 candidate outputs.
|
|
|
|
Approach: for each (field, value) in the canonical record, check whether the
|
|
candidate formatted string contains `str(value)` as a case-sensitive
|
|
substring. This is deliberately simpler than parsing the candidate — the
|
|
linter is responsible for structural checks (punctuation, ordering, italics),
|
|
so the diff only answers: "did the right facts survive the reformat?"
|
|
|
|
Frozen harness module. Edits here invalidate prior scores.
|
|
"""
|
|
|
|
from __future__ import annotations
|
|
|
|
from dataclasses import dataclass, field
|
|
|
|
|
|
@dataclass
|
|
class DiffResult:
|
|
matched: list[str] = field(default_factory=list)
|
|
missing: list[str] = field(default_factory=list)
|
|
|
|
@property
|
|
def total(self) -> int:
|
|
return len(self.matched) + len(self.missing)
|
|
|
|
@property
|
|
def rate(self) -> float:
|
|
if self.total == 0:
|
|
return 1.0
|
|
return len(self.matched) / self.total
|
|
|
|
|
|
def field_diff(candidate: str, canonical: dict) -> DiffResult:
|
|
"""Compare a formatted candidate string against a canonical field dict.
|
|
|
|
Each canonical field counts as matched iff its stringified value appears
|
|
as a case-sensitive substring of `candidate`.
|
|
"""
|
|
result = DiffResult()
|
|
for key, value in canonical.items():
|
|
if str(value) in candidate:
|
|
result.matched.append(key)
|
|
else:
|
|
result.missing.append(key)
|
|
return result
|