Quality checks

DataLeaf validates the generated production site and independent consuming sites.

The architecture fixtures exercise empty sites, inferred and explicit main sections, custom taxonomies and permalinks, nested menus, reordered and disabled outputs, missing optional parameters, language subdirectories and separate domains, regional translation overrides, included and global resources, and math rendered inside shortcodes. They validate generated HTML and links and retain an architecture.json report beside the build manifest.

Two parallel validate-theme jobs build with pinned standard Hugo 0.158.0 and 0.167.0 binaries whose SHA-256 checksums are verified. A pinned Go toolchain exercises local Hugo Module imports. Each job builds the following fixtures:

• root-domain and GitLab Pages/subpath builds of the example;
• clean theme, disabled-search, and module/override consumers;
• the architecture scenarios;
• the visual fixture.

Each job then validates the generated HTML and internal links and fragments, including minified and same-origin absolute URLs. The current-version job also runs the regression tests and the distribution checks once, since they do not depend on the Hugo version; failing tests appear in the merge request’s test report. Logs and generated sites are retained even on failure.

quality-site runs the browser checks on those artifacts without rebuilding anything. Playwright and axe check English/Portuguese pages at 320px and desktop widths in explicit light, explicit dark, and automatic dark modes; see Accessibility checks. Browser assertions cover search, image decoding, horizontal overflow, mobile TOC navigation, Unicode anchors, clipboard fallback, theme persistence, and no-JavaScript keyboard navigation.

Runner minutes are limited, so each check runs where it adds information:

A push to a branch without a merge request runs no pipeline, so opening the MR afterwards does not validate the same commit twice. Use Build > Pipelines > Run pipeline to validate such a branch. Tags run no pipeline, because they point at default-branch commits that were already validated. A newer commit cancels a running pipeline for the same branch or MR, except for the release job.

Jobs have timeouts (10 minutes for builds, 20 for browser checks, 5 for deployment and release), so a hung browser cannot consume an hour of runner time. Each step is a collapsible, timed section of the job log, and a failing job ends by naming the step it failed in.

The HTML validator permits legal minifier output: unquoted attributes, lowercase doctype, void tag syntax, omitted empty body and optional tbody tags, and HTML5 identifiers such as Goldmark’s fn:1. It also allows long document titles used in wrapping checks. Structural, ARIA, and duplicate-ID validation remains active. Regression tests prove malformed nesting and invalid ARIA still fail.

deploy-pages publishes the tested current-Hugo artifact built using CI_PAGES_URL; it does not rebuild or check out the repository. Release publishing is restricted to explicitly requested default-branch web pipelines. Even there, it is a manual job that someone must start after every other stage has passed; see RELEASING.md.

npm run quality:a11y targets WCAG 2.2 AA. For each fixture site it loads home, listing, pagination, article, documentation, taxonomy, archive, and search pages, in English and Portuguese where available. Each page is checked at 320px (the WCAG reflow width) and at desktop width, in explicit light, explicit dark, and automatic dark modes.

The example is built twice, for a root domain and for a subpath. Their pages differ only in URLs, so the deployed Pages build gets this full matrix and the other build gets a URL-shape smoke test. That test covers the home page, an article, and both search pages at desktop width, plus the per-site navigation, theme, motion, and no-JavaScript checks.

Every view runs:

• axe-core with the wcag2a, wcag2aa, wcag21a, wcag21aa, wcag22a, wcag22aa, and best-practice tags. No rule is disabled.
• DataLeaf audits from scripts/lib/a11y.mjs, for problems axe-core does not measure:
  • Symbol-only controls, such as # permalinks, must reach 3:1 contrast including inherited opacity. axe’s contrast rule skips punctuation-only text.
  • Headings must not contain controls that change their accessible name.
  • Current navigation items must differ from their peers by more than color.
  • Screen-reader-only text must stay exposed to assistive technology while occupying no space.
• A keyboard traversal of the whole page. The skip link must be the first stop, and every stop must show a focus indicator and remain visible and unobscured. Focus must eventually leave the page.

Behavioral assertions also cover:

• the search landmark, result headings, and debounced result announcements;
• heading-copy announcements;
• keyboard operation of the theme selector and its visible label within its accessible name;
• reduced motion (no smooth scrolling, transitions, or animations);
• TOC destinations whose heading names match the TOC entries;
• the no-JavaScript skip link, TOC, search, and theme fallbacks.

Regression tests show that each DataLeaf audit fails on known-bad markup.

axe-core’s experimental rules are not run. Its focus-order-semantics rule would require every focusable scrolling equation, table, and code block to be a region landmark, which floods landmark navigation on technical articles. DataLeaf exposes those regions as named groups instead. axe’s incomplete (“needs review”) results are not failures. They come from MathML annotations, aria-hidden decorative glyphs, and hero text on a subtle gradient, which axe cannot sample. That hero text measures at least 4.7:1 at both gradient stops in both themes.

The report, including focus-stop counts and any violations, is written to accessibility.json beside the build manifest.

Install Hugo 0.158.0 or later, Go 1.22 or later, and Node.js 24.8 or later:

npm ci --ignore-scripts
npx playwright install chromium
npm test
npm run build:fixtures

Set SITE_MANIFEST to the path printed by the build (for example, build/0.167.0/sites.json) and run npm run quality. In PowerShell:

$env:SITE_MANIFEST = "build/0.167.0/sites.json"
npm run quality

In a POSIX shell:

SITE_MANIFEST=build/0.167.0/sites.json npm run quality

HUGO_BIN selects a particular Hugo executable. PLAYWRIGHT_CHANNEL=msedge can use an installed Microsoft Edge browser locally; CI uses the Chromium version bundled with its matching Playwright image. Individual checks remain available as quality:html, quality:links, quality:a11y, and quality:distribution. check:hugo-themes is a read-only pre-submission check of the published repository; it needs network access, Go, and Hugo, and is not part of CI.

The complete quality command requires SITE_MANIFEST. Individual HTML, link, and accessibility checks can also use exampleSite/public/ and https://example.com/ without a manifest. Set SITE_BASE_URL when testing another base URL.

To refresh distribution previews after a reviewed visual change, run npm run build:fixtures, then npm run preview:capture and npm run quality:distribution with SITE_MANIFEST set. Without it, the capture uses exampleSite/public, which may be an old build. Captures are actual browser output: the screenshot is the featured article at 1500×1000, showing the rail, type, mathematics, and code; the thumbnail is the front page at 600×400 scaled to 900×600, so its title stays legible at the directory’s smaller sizes.

Automated checks do not replace manual screen-reader, visual, and release-installation review.

npm run quality:visual requires SITE_MANIFEST from build:fixtures. It checks the Pages example and an independent visual fixture at 320, 375, 768, 1024, 1280, 1440, and 1920px in light and dark modes, and the fixture again as a right-to-left site with a second, long-named language at 320, 768, 1024, and 1440px. Coverage includes home/list/pagination views, taxonomies, archives, search, documentation, long titles, three levels of nested menus, nested sections, many and long tags, empty sections, undated entries, all heading levels, numbered/highlighted code, compact/wide tables, equations, small figures, and scientific shortcodes.

The browser checks the manuscript layout (title over subtitle over metadata, paragraphs within the measure, an unboxed table of contents, and from 1152px a side rail beside the text with a sticky table of contents and wide content extending past the measure; one column below), the front page (the latest writing spans the page, titles step down from the page heading to the featured and listed entries, two columns from 1024px with at least three entry titles in the first viewport, one column below, and no featured entry on later pages), discovery pages (titles over entry titles, entry titles aligned with the page title and dates and metadata in the rail from 1152px, the date beside the title below that and above it on phones, no pills or cards, topics in alphabetical columns with tabular counts, archive years beside or over their entries, the search field’s 3:1 rule, both pagination ends, a subsection’s link to its section, an empty section’s notice, and documentation contents on one page), the masthead hierarchy (site title over navigation over the muted site tools, with the example’s navigation on one row from 1024px), page and navigation bounds, line-number alignment, intrinsic figure size, keyboard scrolling/focus, TOC destinations, enlarged text/spacing, and no-JavaScript navigation. On the right-to-left build the same layout checks run in logical terms (the rail, dates, and archive years sit at the right), the site title starts at the right, pagination arrows are mirrored, and code and equations still read left to right. A forced-colors pass checks that focus outlines, current navigation, rules, marks, highlighted code lines, and unavailable pagination stay visible. Bounds failures name the element that escapes the viewport.

A realistic scientific article in the fixture combines several equations, nested and loose lists, a long numbered code block, wide tables, transparent and raster figures, repeated footnotes, theorem and proof blocks, long headings, and mixed code and math. For every width and mode, the browser checks the following:

• Styled letters render as Unicode (ℝ, 𝔼, 𝒩) and math uses a MATH-table font stack.
• The bundled Source Serif 4, IBM Plex Sans, IBM Plex Mono, and STIX Two Math faces load, and prose is set in the serif.

A type specimen page checks the design system at every width and mode: no shadows, square content corners and small control radii, each typeface role, an ordered heading scale, paragraph spacing equal to the body size, prose within the reading measure, and the link, annotation, and mark colors. It also gets an axe scan. The regression tests keep the design tokens reference in sync with tokens.css, check that components use only defined tokens, and check WCAG AA contrast for every color pair in both schemes.

• Aligned equations line up at their alignment points, and equation regions never grow wider than four times the column.
• Math renders in the TOC, statement titles, and permalink names.
• Transparent figures keep a light backdrop, and raster figures reserve their intrinsic size.
• Every line number sits beside its line.
• Footnotes, task lists, and glossaries are styled as intended.

Both technical articles run axe with WCAG 2.2 A/AA and best-practice rules in every width/mode combination.

At 200% text on a 320px screen, the enlarged-text check tolerates a page widened by inline math, and nothing else: browsers cannot wrap an inline MathML formula. visual.json records completed views; a failure capture is retained in visual-captures/.

Set VISUAL_SCREENSHOTS=1 to save full-page captures for every view. These are review artifacts, not pixel-perfect golden-image tests; operating-system font rendering can differ. CI runs the checks on both supported Hugo versions and retains the reports and any failure captures.