Skip to content

CLAUDE.md

Latest commit

 

History

History
204 lines (146 loc) · 14.9 KB

CLAUDE.md

File metadata and controls

204 lines (146 loc) · 14.9 KB
Raw

CLAUDE.md

This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.

Core Working Principles

Adapted from Andrej Karpathy's four CLAUDE.md principles — read these before touching code. They target reasoning failures (wrong assumptions, over-engineering, scope creep, weak success criteria), not formatting.

  1. Think Before Coding — Don't assume. Don't hide confusion. Surface tradeoffs.

    • State assumptions explicitly; if uncertain, ask rather than guess.
    • Present competing interpretations instead of silently picking one.
    • Call out inconsistencies, confusion, and tradeoffs as you find them.

  2. Simplicity First — Minimum code that solves the problem. Nothing speculative.

    • No features beyond what was asked.
    • No abstractions for single-use code.
    • No "flexibility" or configurability that wasn't requested.
    • No error handling for scenarios that cannot occur.

  3. Surgical Changes — Touch only what you must. Clean up only your own mess.

    • Match the surrounding style and conventions.
    • Don't refactor unbroken adjacent code that's orthogonal to the task.
    • Only remove what your change rendered obsolete.

  4. Goal-Driven Execution — Define success criteria. Loop until verified.

    • Turn tasks into verifiable goals: "add validation" → "write tests, then make them pass"; "fix the bug" → "reproduce it in a test, then fix"; "refactor X" → "ensure tests pass before and after".
    • Iterate against those criteria instead of asking for constant clarification.

Setup After Clone

git config core.hooksPath .githooks               # enable pre-commit hook (fmt + clippy + tests)

Build & Test Commands

cargo fmt                                         # auto-format code (always run before build)
cargo build                                       # debug build
cargo build --release                             # release build (~11s)
cargo test                                        # unit tests + offline integration tests (~300 tests)
cargo test --test integration_crawl -- --ignored --test-threads=1  # network integration tests (crawls crawler.siteone.io)
cargo test scoring::ci_gate::tests::all_checks_pass  # run a single test by name
cargo clippy -- -D warnings                       # lint (CI enforces zero warnings)
cargo fmt -- --check                              # format check

# Browser rendering (`--browser`, chromiumoxide/CDP) is a DEFAULT feature — `cargo build`,
# `cargo test`, `cargo clippy` all include it. Build/test the lean variant (no chromiumoxide):
cargo build --release --no-default-features        # lean build, ~6 MB smaller, no browser
cargo test --no-default-features                   # tests without the browser feature
cargo clippy --no-default-features -- -D warnings  # lint the lean variant

Quick Run

./target/release/siteone-crawler --url=https://example.com --single-page
./target/release/siteone-crawler --url=https://example.com --output=json --http-cache-dir=  # no cache
./target/release/siteone-crawler --html-to-markdown=page.html                               # convert local HTML to markdown (stdout)
./target/release/siteone-crawler --html-to-markdown=page.html --html-to-markdown-output=page.md  # convert to file

Architecture

Crawl Lifecycle (in order)

  1. CLI Parsing (Initiator → CoreOptions::parse_argv()): Parses 120+ CLI options, merges config file if present, validates. Exits with code 101 on error, code 2 on --help/--version. Non-crawl utility modes (--serve-markdown, --serve-offline, --html-to-markdown) exit early in main.rs before creating the Manager.

  2. Analyzer Registration (Initiator::register_analyzers()): Creates all 17 analyzer instances (Accessibility, BestPractice, BrowserConsole, Caching, ContentType, DNS, ExternalLinks, Fastest, Headers, Page404, Redirects, Security, SeoAndOpenGraph, SkippedUrls, Slowest, SourceDomains, SslTls) and registers them with AnalysisManager. Some analyzers receive config from CLI options (e.g. fastest_top_limit, max_heading_level); BrowserConsole is active only in --browser mode.

  3. Manager Setup (Manager::run()): Creates Status (result storage), Output (text/json/multi), HttpClient (with optional proxy, auth, cache), ContentProcessorManager (HTML, CSS, JS, XML, Astro, Next.js, Svelte processors), and the Crawler instance. The crawl loop fetches through an Arc<dyn Fetcher> (src/engine/fetcher.rs): by default the HttpClient; with the browser feature + --browser, a BrowserRenderer that renders each HTML page in Chromium and returns the same HttpResponse (plus browser_diagnostics). Everything downstream is unchanged.

  4. Robots.txt Fetch (Crawler::fetch_robots_txt()): Before crawling starts, fetches and parses /robots.txt from the initial domain. Respects --ignore-robots-txt option.

  5. Crawl Loop (Crawler::run()): Breadth-first concurrent URL processing:

    • URL queue (DashMap) seeded with initial URL
    • Tokio tasks limited by Semaphore (= --workers count) + rate limiting (--max-reqs-per-sec)
    • Per-URL flow: check robots.txt → HTTP request → on error, store with negative status code → on success, run content processors → extract links from HTML → enqueue discovered URLs
    • Content processors (HtmlProcessor, CssProcessor, etc.) transform response bodies during crawl — used by offline/markdown exporters for URL rewriting
    • Each visited URL's response is stored in Status for post-crawl analysis
    • Per-URL data collected: status code, headers, body, response time, content type, size, redirects

  6. Post-Crawl Analysis (Manager::run_post_crawl()): Sequential pipeline after crawling ends:

    • Transfer skipped URLs from crawler to Status
    • Run all registered analyzers (AnalysisManager::run_analyzers()): each analyzer gets read access to Status (all crawled data) and write access to Output (adds tables/findings)
    • Add content processor stats table

  7. Exporters (Manager::run_exporters()): Generate output files based on CLI options:

    • SitemapExporter: XML/TXT sitemap files
    • OfflineWebsiteExporter: Static website copy with rewritten relative URLs
    • MarkdownExporter: HTML→Markdown conversion with relative .md links
    • FileExporter: Save text/JSON output to file
    • HtmlReport: Self-contained HTML report (also used by Mailer and Upload)
    • MailerExporter: Email HTML report via SMTP
    • UploadExporter: Upload report to remote server
    • AnimationExporter (feature browser): Assemble per-page screenshots into a GIF/MP4 animation (GIF via the embedded image crate, MP4 via external ffmpeg)

  8. Scoring (scorer::calculate_scores()): Computes quality scores (0–10) across 5 weighted categories (Performance 20%, SEO 20%, Security 25%, Accessibility 20%, Best Practices 15%). Deductions come from summary findings (criticals, warnings) and stats (404s, 5xx, slow responses).

  9. CI/CD Gate (ci_gate::evaluate()): When --ci is active, checks scores and stats against configurable thresholds (--ci-min-score, --ci-max-404, etc.). Returns exit code 10 on failure.

  10. Summary & Output (Output::add_summary(), Output::end()): Prints summary table with OK/Warning/Critical counts, finalizes output. Exit code: 0 = success, 3 = no pages crawled, 10 = CI gate failed.

How Analyzers Work

Each analyzer implements the Analyzer trait (analysis/analyzer.rs). Analyzers are post-crawl only — they don't run during crawling. The AnalysisManager calls each analyzer's analyze(&Status, &mut Output) method after all URLs have been visited. Analyzers read crawled data from Status (visited URLs, response headers, bodies, skipped URLs) and produce SuperTable instances that get added to Output. Analyzers also add Item entries to the Summary (OK, Warning, Critical, Info findings) which feed into scoring.

How Content Processors Work

Content processors implement ContentProcessor (content_processor/content_processor.rs) and run during crawl on each URL's response body. They serve two purposes: (1) transform content for offline/markdown export (rewrite URLs to relative paths), and (2) extract metadata (links, assets). Processors are type-specific: HtmlProcessor handles HTML, CssProcessor handles CSS url() references, etc. The ContentProcessorManager dispatches to the right processor based on content type.

Concurrency Model

The crawler uses tokio for async I/O with a semaphore-based worker pool (options.workers). Shared state uses:

  • Arc<DashMap<...>> for lock-free concurrent maps (URL queue, visited URLs, skipped URLs)
  • Arc<Mutex<...>> for sequential-access state (Status, Output, AnalysisManager)
  • Arc<AtomicBool/AtomicUsize> for simple flags and counters

Key Traits

  • Analyzer (analysis/analyzer.rs): Post-crawl analysis (SEO, security, headers, etc.). Each analyzer gets &Status and &mut Output.
  • Exporter (export/exporter.rs): Output generators (HTML report, offline website, markdown, sitemap, mailer, upload).
  • Output (output/output.rs): Formatting backend. Implementations: TextOutput, JsonOutput, MultiOutput.
  • ContentProcessor (content_processor/content_processor.rs): Per-URL content transformation during crawl (HTML, JS, CSS, XML processors).

Options System

CLI options are defined in options/core_options.rs via get_options() which returns an Options struct with typed option groups. Parsing flow: parse_argv() → merge config file → parse flags → CoreOptions::from_options() → apply_option_value() for each option. New CLI options require: adding the field to CoreOptions, a case in apply_option_value(), and an entry in the appropriate option group.

Exit Codes

Code Meaning
0 Success (with --ci: all thresholds passed)
1 Runtime error
2 Help/version displayed
3 No pages successfully crawled (DNS failure, timeout, etc.)
10 CI/CD quality gate failed
101 Configuration error

HTTP Response Body

HttpResponse.body is Option<Vec<u8>> (not String) to preserve binary data for images, fonts, etc. Use body_text() for string content. Failed HTTP requests return Ok(HttpResponse) with negative status codes (-1 connection error, -2 timeout, -4 send error), not Err.

Testing Structure

  • Unit tests: In-file #[cfg(test)] mod tests blocks (standard Rust convention)
  • Integration tests: tests/integration_crawl.rs with shared helpers in tests/common/mod.rs
  • Network-dependent integration tests are #[ignore] — run explicitly with --ignored

Testing Complex Scenarios with Sample Websites

The crawler has a built-in HTTP server (--serve-offline=<dir>) that can serve any local directory as a static website. This enables efficient local testing of edge cases without deploying a real site:

  1. Create a sample website directory, e.g. ./tmp/sample-website-xyz/
  2. Add HTML files and assets simulating the desired scenario (spaces in filenames, special characters, redirect chains, broken links, specific heading structures, etc.)
  3. Start the built-in server: ./target/release/siteone-crawler --serve-offline=./tmp/sample-website-xyz/ --serve-port=8888
  4. In another terminal, crawl the local site: ./target/release/siteone-crawler --url=http://127.0.0.1:8888/
  5. Verify the crawler handles the scenario correctly (output, offline export, analysis results)

This approach is useful for reproducing bug reports, testing regex edge cases (e.g. URLs with spaces, HTML entities, unusual attribute quoting), validating offline/markdown export for specific HTML structures, and any scenario that would be hard to find on a live website.

Key Files

  • src/engine/crawler.rs (~1700 lines): Core crawl loop, URL queue management, HTML/content parsing
  • src/options/core_options.rs (~2500 lines): All 120+ CLI options, parsing, validation
  • src/export/utils/offline_url_converter.rs (~1400 lines): URL-to-file-path conversion for offline export
  • src/export/html_report/report.rs: HTML report generation with embedded template
  • src/scoring/scorer.rs: Quality score calculation from summary findings
  • src/scoring/ci_gate.rs: CI/CD threshold evaluation
  • src/engine/fetcher.rs: Fetcher trait — the single seam the crawl loop fetches through; HttpClient (direct HTTP) and BrowserRenderer both implement it
  • src/browser/ (feature browser): BrowserRenderer (renderer.rs), Chromium detection/download/launch (launcher.rs), CDP diagnostics collection (diagnostics.rs), screenshots + pre-capture animation settling (screenshot.rs), cookie-banner dismissal/hiding (cookie_consent.rs). diagnostics.rs data types are always compiled so HttpResponse can carry an inert Option<BrowserDiagnostics>
  • src/export/animation_exporter.rs (feature browser): builds GIF/MP4 animations from per-page screenshots (GIF via the embedded image crate, MP4 via external ffmpeg; frames streamed to disk for O(1) memory)
  • src/analysis/browser_console_analyzer.rs: reports browser console/JS/network/security diagnostics (the browser-mode analyzer; active only in --browser mode)

Edition & Rust Version

Project uses edition = "2024" (Rust 1.85+) with rust-version = "1.94". Edition 2024 features used throughout: unsafe extern blocks, if let chaining (if let ... && ...), unsafe { std::env::set_var() }.

Commit Policy

Never commit automatically. Commits are only allowed on explicit user request. Before every commit, always run git status, review the changes, and stage only the relevant files — never use git add -A or git add . blindly.

Commit Messages

Use Conventional Commits: feat:, fix:, refactor:, perf:, docs:, style:, ci:, chore:, test:. Examples:

  • feat: add built-in HTTP server for markdown/offline exports
  • fix: correct non-ASCII text corruption in heading ID generation
  • perf: eliminate heap allocation in content_type_for_extension
  • chore: bump version to 2.0.3

Releasing a New Version

  1. Update version in Cargo.toml (version = "X.Y.Z")
  2. Update version in src/version.rs (pub const CODE: &str = "X.Y.Z.YYYYMMDD";)
  3. Run cargo check so that Cargo.lock is updated with the new version
  4. Commit all three files (Cargo.toml, src/version.rs, Cargo.lock): git commit -m "chore: bump version to X.Y.Z"
  5. Tag and push: git tag vX.Y.Z && git push && git push --tags

Important Conventions

  • Tables, column order, and formatting must stay consistent across versions. The HTML parser uses the scraper crate.
  • HTTP cache lives in tmp/http-client-cache/ by default. Delete it for fresh crawls or use --http-cache-dir= to disable.
  • rustls requires explicit ring CryptoProvider installation in main.rs.