ocertat/cmos
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity
Files
main
cmos/tests/test_linter.py
T
cmos dev fe5e071066 linter: expand to 21 rules covering all 8 source types (v0.3.0) 
Close the defense-in-depth gap flagged after iter 6: chapter, magazine,
newspaper, social_media, podcast, and video exemplars had ZERO applicable
linter rules, so their 100% linter pass rate was trivially true. Every
exemplar now has 3-6 structural rules firing.

Bump LINTER_VERSION to v0.3.0. Per harness discipline, this invalidates
prior run logs' scores for cross-version comparison (they stay in the
log as history). The re-baseline run still scores scalar 1.000, canary
1.000 on all 14 exemplars — formatter was already producing output that
matches the new structural conventions, so tightening the linter didn't
surface any regressions.

New rules (12):
 - chapter_book_title_italicized
 - chapter_in_book_marker
 - magazine_name_italicized
 - newspaper_name_italicized
 - periodical_comma_before_date  (magazine + newspaper; not journal)
 - social_media_post_quoted
 - social_media_platform_comma_date
 - podcast_series_italicized
 - podcast_episode_quoted
 - podcast_format_label
 - video_title_quoted             (in quotes, NOT italicized)
 - video_format_label

Extended:
 - article_title_quoted           now covers journal, magazine, newspaper,
                                  and accepts ?/! as terminal punctuation

Each rule has a positive + negative unit test with the CMOS/Purdue source
cited in the docstring. Tests: 48 → 78 (30 new).

Per-exemplar applicable-rule counts after this change:
 book    4    journal  5-6  web        3
 chapter 4    magazine 5    social     4
 podcast 5    video    4    newspaper  5
2026-04-10 22:01:56 -04:00

603 lines
21 KiB
Python
Raw Permalink Blame History

This file contains ambiguous Unicode characters
This file contains Unicode characters that might be confused with other characters. If you think that this is intentional, you can safely ignore this warning. Use the Escape button to reveal them.
"""Tests for src/cmos/linter.py — deterministic CMOS 18 structural rules.
Each rule has a positive and a negative unit test. Doc comments cite the
source that justifies the rule (Purdue OWL page or CMOS quick guide URL).
The linter only checks structural properties of the RAW output string
(italic markers as emitted, punctuation as emitted). Fact survival
(author/title/year/etc. matching canonical) is the diff's job, not the
linter's, so we deliberately don't duplicate it here.
"""
from cmos.linter import (
LINTER_VERSION,
check_passes,
lint,
rule_article_title_quoted,
rule_book_title_italicized,
rule_chapter_book_title_italicized,
rule_chapter_in_book_marker,
rule_doi_is_https_url,
rule_ends_with_period,
rule_journal_title_italicized,
rule_magazine_name_italicized,
rule_newspaper_name_italicized,
rule_no_ibid,
rule_no_place_of_publication_for_books,
rule_page_range_uses_en_dash,
rule_periodical_comma_before_date,
rule_podcast_episode_quoted,
rule_podcast_format_label,
rule_podcast_series_italicized,
rule_social_media_platform_comma_date,
rule_social_media_post_quoted,
rule_video_format_label,
rule_video_title_quoted,
rule_web_page_title_quoted,
)
from harness.score import Exemplar
BOOK = Exemplar(
name="book",
source="test",
type="book",
tags=["book"],
canary=False,
messy_input="",
canonical={
"author": "Yu, Charles",
"title": "Interior Chinatown",
"publisher": "Pantheon Books",
"year": 2020,
},
expected_bibliography="Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.",
)
JOURNAL = Exemplar(
name="journal",
source="test",
type="journal",
tags=["journal", "doi"],
canary=False,
messy_input="",
canonical={
"author": "Kwon, Hyeyoung",
"article_title": "Inclusion Work: Children of Immigrants Claiming Membership in Everyday Life",
"journal": "American Journal of Sociology",
"volume": 127,
"issue": 6,
"year": 2022,
"pages": "1818–59",
"doi": "https://doi.org/10.1086/720277",
},
expected_bibliography=(
'Kwon, Hyeyoung. "Inclusion Work: Children of Immigrants Claiming Membership in Everyday Life." '
"*American Journal of Sociology* 127, no. 6 (2022): 1818–59. https://doi.org/10.1086/720277."
),
)
WEB = Exemplar(
name="web",
source="test",
type="web",
tags=["web"],
canary=False,
messy_input="",
canonical={
"author": "Google",
"title": "Privacy Policy",
"site": "Privacy & Terms",
"date": "November 15, 2023",
"url": "https://policies.google.com/privacy",
},
expected_bibliography='Google. "Privacy Policy." Privacy & Terms. Effective November 15, 2023. https://policies.google.com/privacy.',
)
CHAPTER = Exemplar(
name="chapter",
source="test",
type="chapter",
tags=["chapter"],
canary=False,
messy_input="",
canonical={
"author": "Doyle, Kathleen",
"chapter_title": "The Queen Mary Psalter",
"book_title": "The Book by Design: The Remarkable Story of the World's Greatest Invention",
"editors": "edited by P. J. M. Marks and Stephen Parkin",
"publisher": "University of Chicago Press",
"year": 2023,
},
expected_bibliography='Doyle, Kathleen. "The Queen Mary Psalter." In *The Book by Design: The Remarkable Story of the World\'s Greatest Invention*, edited by P. J. M. Marks and Stephen Parkin. University of Chicago Press, 2023.',
)
MAGAZINE = Exemplar(
name="magazine",
source="test",
type="magazine",
tags=["magazine"],
canary=False,
messy_input="",
canonical={
"author": "Mead, Rebecca",
"article_title": "Terms of Aggrievement",
"magazine": "New Yorker",
"date": "December 18, 2023",
},
expected_bibliography='Mead, Rebecca. "Terms of Aggrievement." *New Yorker*, December 18, 2023.',
)
NEWSPAPER = Exemplar(
name="newspaper",
source="test",
type="newspaper",
tags=["newspaper"],
canary=False,
messy_input="",
canonical={
"author": "Blum, Dani",
"article_title": "Are Flax Seeds All That?",
"newspaper": "New York Times",
"date": "December 13, 2023",
"url": "https://www.nytimes.com/2023/12/13/well/eat/flax-seeds-benefits.html",
},
expected_bibliography='Blum, Dani. "Are Flax Seeds All That?" *New York Times*, December 13, 2023. https://www.nytimes.com/2023/12/13/well/eat/flax-seeds-benefits.html.',
)
SOCIAL = Exemplar(
name="social_media",
source="test",
type="social_media",
tags=["social_media"],
canary=False,
messy_input="",
canonical={
"author": "Chicago Manual of Style",
"post_text": "Is the world ready for singular they? We thought so in 1993.",
"platform": "Facebook",
"date": "April 17, 2015",
"url": "https://www.facebook.com/ChicagoManual/posts/10152906193679151",
},
expected_bibliography='Chicago Manual of Style. "Is the world ready for singular they? We thought so in 1993." Facebook, April 17, 2015. https://www.facebook.com/ChicagoManual/posts/10152906193679151.',
)
PODCAST = Exemplar(
name="podcast",
source="test",
type="podcast",
tags=["podcast"],
canary=False,
messy_input="",
canonical={
"host": "Ober, Lauren, host",
"podcast": "The Loudest Girl in the World",
"season_episode": "Season 1, episode 2",
"episode_title": "Goodbye, Routine; Hello, Meltdown!",
"publisher": "Pushkin Industries",
"date": "September 13, 2022",
"format": "Podcast",
"duration": "41 min., 37 sec.",
"url": "https://www.pushkin.fm/podcasts/loudest-girl-in-the-world",
},
expected_bibliography='Ober, Lauren, host. *The Loudest Girl in the World*. Season 1, episode 2, "Goodbye, Routine; Hello, Meltdown!" Pushkin Industries, September 13, 2022. Podcast, 41 min., 37 sec. https://www.pushkin.fm/podcasts/loudest-girl-in-the-world.',
)
VIDEO = Exemplar(
name="video",
source="test",
type="video",
tags=["video"],
canary=False,
messy_input="",
canonical={
"author": "Cowan, Vaitea",
"title": "How Green Hydrogen Could End the Fossil Fuel Era",
"venue": "TED Talk, Vancouver, BC",
"date": "April 2022",
"format": "Video",
"duration": "9 min., 15 sec.",
"url": "https://www.ted.com/talks/vaitea_cowan_how_green_hydrogen_could_end_the_fossil_fuel_era",
},
expected_bibliography='Cowan, Vaitea. "How Green Hydrogen Could End the Fossil Fuel Era." TED Talk, Vancouver, BC, April 2022. Video, 9 min., 15 sec. https://www.ted.com/talks/vaitea_cowan_how_green_hydrogen_could_end_the_fossil_fuel_era.',
)
def test_linter_version_is_set():
assert LINTER_VERSION
assert isinstance(LINTER_VERSION, str)
def test_ends_with_period_passes_on_well_formed_entry():
# CMOS 18: bibliography entries separate major elements by periods and
# end with a period. Source: chicagomanualofstyle.org quick guide.
result = rule_ends_with_period("Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.", BOOK)
assert result.applicable is True
assert result.passed is True
def test_ends_with_period_fails_on_missing_period():
result = rule_ends_with_period("Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020", BOOK)
assert result.applicable is True
assert result.passed is False
# --- No place of publication for books (CMOS 18 change from 17, CMOS 14.30) ----
def test_no_place_of_publication_passes_when_absent():
# CMOS 18: place of publication no longer required for books.
# Source: CMOS 14.30 (cited on chicagomanualofstyle.org quick guide).
result = rule_no_place_of_publication_for_books(
"Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.", BOOK
)
assert result.applicable is True
assert result.passed is True
def test_no_place_of_publication_fails_when_city_prefix_present():
result = rule_no_place_of_publication_for_books(
"Yu, Charles. *Interior Chinatown*. New York: Pantheon Books, 2020.", BOOK
)
assert result.passed is False
def test_no_place_of_publication_not_applicable_to_journals():
# The rule should not fire on non-book types even if their output happens
# to contain a colon (which journals routinely do, e.g., article title).
result = rule_no_place_of_publication_for_books("anything: anything.", JOURNAL)
assert result.applicable is False
# --- Book title italicized ----------------------------------------------------
def test_book_title_italicized_passes_when_wrapped_in_asterisks():
# CMOS 18 bibliography basics: titles of books and journals are italicized.
# Source: chicagomanualofstyle.org quick guide; Purdue OWL CMOS 18 NB poster
# page 3 "Bibliography: Basics".
result = rule_book_title_italicized(
"Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.", BOOK
)
assert result.passed is True
def test_book_title_italicized_fails_when_title_plain():
result = rule_book_title_italicized(
"Yu, Charles. Interior Chinatown. Pantheon Books, 2020.", BOOK
)
assert result.passed is False
def test_book_title_italicized_not_applicable_to_web():
result = rule_book_title_italicized("whatever", WEB)
assert result.applicable is False
# --- No "Ibid." (CMOS 18 deprecation) -----------------------------------------
def test_no_ibid_passes_on_clean_entry():
# CMOS 18 deprecates "Ibid." in favor of shortened notes.
# Source: Widener "New in 18th" LibGuide.
result = rule_no_ibid("Yu, Charles. *Interior Chinatown*. Pantheon Books, 2020.", BOOK)
assert result.passed is True
def test_no_ibid_fails_when_ibid_present():
result = rule_no_ibid("Ibid., 42.", BOOK)
assert result.passed is False
# --- Journal title italicized --------------------------------------------------
def test_journal_title_italicized_passes_when_wrapped():
result = rule_journal_title_italicized(JOURNAL.expected_bibliography, JOURNAL)
assert result.passed is True
def test_journal_title_italicized_fails_when_plain():
bad = JOURNAL.expected_bibliography.replace(
"*American Journal of Sociology*", "American Journal of Sociology"
)
result = rule_journal_title_italicized(bad, JOURNAL)
assert result.passed is False
# --- Article title in quotes ---------------------------------------------------
def test_article_title_quoted_passes_when_wrapped_in_double_quotes():
# CMOS 18: titles of articles, chapters, poems are in quotation marks.
# Source: Purdue OWL CMOS 18 NB poster page 3 Bibliography Basics.
result = rule_article_title_quoted(JOURNAL.expected_bibliography, JOURNAL)
assert result.passed is True
def test_article_title_quoted_fails_when_unquoted():
bad = JOURNAL.expected_bibliography.replace(
'"Inclusion Work: Children of Immigrants Claiming Membership in Everyday Life."',
"Inclusion Work: Children of Immigrants Claiming Membership in Everyday Life.",
)
result = rule_article_title_quoted(bad, JOURNAL)
assert result.passed is False
# --- Page range uses en-dash ---------------------------------------------------
def test_page_range_uses_en_dash_passes():
# CMOS convention: page ranges use en-dash (–), not ASCII hyphen (-).
result = rule_page_range_uses_en_dash(JOURNAL.expected_bibliography, JOURNAL)
assert result.passed is True
def test_page_range_uses_en_dash_fails_on_ascii_hyphen():
bad = JOURNAL.expected_bibliography.replace("1818–59", "1818-59")
result = rule_page_range_uses_en_dash(bad, JOURNAL)
assert result.passed is False
# --- DOI as https://doi.org/ URL -----------------------------------------------
def test_doi_is_https_url_passes_when_canonical_prefix_present():
# CMOS 18 prefers DOIs formatted as https://doi.org/... rather than bare
# DOIs or "DOI:" prefixes. Source: CMOS 18 quick guide.
result = rule_doi_is_https_url(JOURNAL.expected_bibliography, JOURNAL)
assert result.passed is True
def test_doi_is_https_url_fails_on_bare_doi():
bad = JOURNAL.expected_bibliography.replace("https://doi.org/10.1086/720277", "10.1086/720277")
result = rule_doi_is_https_url(bad, JOURNAL)
assert result.passed is False
def test_doi_rule_not_applicable_without_doi():
# Web exemplar has no DOI field.
result = rule_doi_is_https_url(WEB.expected_bibliography, WEB)
assert result.applicable is False
# --- Web page title in quotes --------------------------------------------------
def test_web_page_title_quoted_passes():
result = rule_web_page_title_quoted(WEB.expected_bibliography, WEB)
assert result.passed is True
def test_web_page_title_quoted_fails_when_unquoted():
bad = WEB.expected_bibliography.replace('"Privacy Policy."', "Privacy Policy.")
result = rule_web_page_title_quoted(bad, WEB)
assert result.passed is False
# --- Article title rule extended to magazines and newspapers -----------------
def test_article_title_quoted_passes_on_magazine():
# Magazines follow the same article-title convention as journals.
result = rule_article_title_quoted(MAGAZINE.expected_bibliography, MAGAZINE)
assert result.applicable is True
assert result.passed is True
def test_article_title_quoted_passes_on_newspaper():
result = rule_article_title_quoted(NEWSPAPER.expected_bibliography, NEWSPAPER)
assert result.applicable is True
assert result.passed is True
def test_article_title_quoted_not_applicable_to_book():
result = rule_article_title_quoted(BOOK.expected_bibliography, BOOK)
assert result.applicable is False
# --- Chapter rules ------------------------------------------------------------
def test_chapter_book_title_italicized_passes():
# Chapters have their enclosing book's title italicized (CMOS 18,
# "Article, Chapter, Essay, etc., in a Book" model).
result = rule_chapter_book_title_italicized(CHAPTER.expected_bibliography, CHAPTER)
assert result.applicable is True
assert result.passed is True
def test_chapter_book_title_italicized_fails_when_plain():
bad = CHAPTER.expected_bibliography.replace(
"*The Book by Design: The Remarkable Story of the World's Greatest Invention*",
"The Book by Design: The Remarkable Story of the World's Greatest Invention",
)
result = rule_chapter_book_title_italicized(bad, CHAPTER)
assert result.passed is False
def test_chapter_book_title_italicized_not_applicable_to_book():
result = rule_chapter_book_title_italicized(BOOK.expected_bibliography, BOOK)
assert result.applicable is False
def test_chapter_in_book_marker_passes():
# Chapter entries include the keyword "In" before the italicized book
# title to distinguish them from standalone books.
result = rule_chapter_in_book_marker(CHAPTER.expected_bibliography, CHAPTER)
assert result.passed is True
def test_chapter_in_book_marker_fails_without_in():
bad = CHAPTER.expected_bibliography.replace(" In *", " *")
result = rule_chapter_in_book_marker(bad, CHAPTER)
assert result.passed is False
# --- Magazine and newspaper name italicized ----------------------------------
def test_magazine_name_italicized_passes():
# CMOS 18: magazine titles are italicized.
result = rule_magazine_name_italicized(MAGAZINE.expected_bibliography, MAGAZINE)
assert result.passed is True
def test_magazine_name_italicized_fails_when_plain():
bad = MAGAZINE.expected_bibliography.replace("*New Yorker*", "New Yorker")
result = rule_magazine_name_italicized(bad, MAGAZINE)
assert result.passed is False
def test_newspaper_name_italicized_passes():
result = rule_newspaper_name_italicized(NEWSPAPER.expected_bibliography, NEWSPAPER)
assert result.passed is True
def test_newspaper_name_italicized_fails_when_plain():
bad = NEWSPAPER.expected_bibliography.replace("*New York Times*", "New York Times")
result = rule_newspaper_name_italicized(bad, NEWSPAPER)
assert result.passed is False
# --- Periodical comma before date (magazine/newspaper only) -------------------
def test_periodical_comma_before_date_passes_on_magazine():
# Magazines and newspapers use a COMMA between the italicized name and
# the date, not a period: *New Yorker*, December 18, 2023.
result = rule_periodical_comma_before_date(MAGAZINE.expected_bibliography, MAGAZINE)
assert result.passed is True
def test_periodical_comma_before_date_fails_when_period_used():
bad = MAGAZINE.expected_bibliography.replace("*New Yorker*,", "*New Yorker*.")
result = rule_periodical_comma_before_date(bad, MAGAZINE)
assert result.passed is False
def test_periodical_comma_before_date_passes_on_newspaper():
result = rule_periodical_comma_before_date(NEWSPAPER.expected_bibliography, NEWSPAPER)
assert result.passed is True
def test_periodical_comma_before_date_not_applicable_to_journal():
# Journals use volume/issue/year in parens, no comma-after-italic pattern.
result = rule_periodical_comma_before_date(JOURNAL.expected_bibliography, JOURNAL)
assert result.applicable is False
# --- Social media rules -------------------------------------------------------
def test_social_media_post_quoted_passes():
# The post content is wrapped in straight double quotes.
result = rule_social_media_post_quoted(SOCIAL.expected_bibliography, SOCIAL)
assert result.passed is True
def test_social_media_post_quoted_fails_when_title_cased():
# Title-casing is a common LLM error on social media content.
bad = SOCIAL.expected_bibliography.replace(
"Is the world ready for singular they? We thought so in 1993.",
"Is the World Ready for Singular They? We Thought So in 1993.",
)
result = rule_social_media_post_quoted(bad, SOCIAL)
assert result.passed is False
def test_social_media_platform_comma_date_passes():
# The platform name is followed by a comma (not a period) before the date.
result = rule_social_media_platform_comma_date(SOCIAL.expected_bibliography, SOCIAL)
assert result.passed is True
def test_social_media_platform_comma_date_fails_when_period_used():
bad = SOCIAL.expected_bibliography.replace("Facebook, April", "Facebook. April")
result = rule_social_media_platform_comma_date(bad, SOCIAL)
assert result.passed is False
# --- Podcast rules ------------------------------------------------------------
def test_podcast_series_italicized_passes():
# Podcast series titles are italicized like book titles.
result = rule_podcast_series_italicized(PODCAST.expected_bibliography, PODCAST)
assert result.passed is True
def test_podcast_series_italicized_fails_when_plain():
bad = PODCAST.expected_bibliography.replace(
"*The Loudest Girl in the World*", "The Loudest Girl in the World"
)
result = rule_podcast_series_italicized(bad, PODCAST)
assert result.passed is False
def test_podcast_episode_quoted_passes():
# Episode titles are in quotes (including terminal punctuation like !).
result = rule_podcast_episode_quoted(PODCAST.expected_bibliography, PODCAST)
assert result.passed is True
def test_podcast_episode_quoted_fails_when_unquoted():
bad = PODCAST.expected_bibliography.replace(
'"Goodbye, Routine; Hello, Meltdown!"',
"Goodbye, Routine; Hello, Meltdown!",
)
result = rule_podcast_episode_quoted(bad, PODCAST)
assert result.passed is False
def test_podcast_format_label_passes():
# CMOS includes a "Podcast," format label before duration.
result = rule_podcast_format_label(PODCAST.expected_bibliography, PODCAST)
assert result.passed is True
def test_podcast_format_label_fails_when_missing():
bad = PODCAST.expected_bibliography.replace("Podcast, ", "")
result = rule_podcast_format_label(bad, PODCAST)
assert result.passed is False
# --- Video rules --------------------------------------------------------------
def test_video_title_quoted_passes():
# Video/TED Talk titles are in quotes (not italicized).
result = rule_video_title_quoted(VIDEO.expected_bibliography, VIDEO)
assert result.passed is True
def test_video_title_quoted_fails_when_italicized():
bad = VIDEO.expected_bibliography.replace(
'"How Green Hydrogen Could End the Fossil Fuel Era."',
"*How Green Hydrogen Could End the Fossil Fuel Era.*",
)
result = rule_video_title_quoted(bad, VIDEO)
assert result.passed is False
def test_video_format_label_passes():
# CMOS includes a "Video," format label before duration.
result = rule_video_format_label(VIDEO.expected_bibliography, VIDEO)
assert result.passed is True
def test_video_format_label_fails_when_missing():
bad = VIDEO.expected_bibliography.replace("Video, ", "")
result = rule_video_format_label(bad, VIDEO)
assert result.passed is False
# --- Aggregate check_passes ---------------------------------------------------
def test_check_passes_on_canonical_outputs():
# Every in-test exemplar's expected_bibliography must lint clean.
# If any fail, either the linter is too strict or the exemplar is wrong —
# either way, we want to know immediately.
for ex in (BOOK, JOURNAL, WEB, CHAPTER, MAGAZINE, NEWSPAPER, SOCIAL, PODCAST, VIDEO):
assert check_passes(ex.expected_bibliography, ex), f"linter rejected {ex.name}"
Reference in New Issue View Git Blame Copy Permalink