✨ Preprocessor-aware C/C++ extraction via libclang

docs/source/components/analyse.rst

@@ -210,3 +210,84 @@ This single comment line creates a complete **Sphinx-Needs** item equivalent to:
    .. impl:: Function Implementation
        :id: IMPL_001
        :links: REQ_001, REQ_002
+
+.. _preprocessor_engine:
+
+Preprocessor-Aware C/C++ Extraction (libclang)
+----------------------------------------------
+
+By default, **Source Analyse** uses a tree-sitter parser that extracts **every** comment,
+regardless of the C preprocessor. For C/C++ projects that rely on conditional compilation
+(``#ifdef VARIANT_A`` …), this means needs from *all* branches are extracted — even
+branches that are never compiled.
+
+The optional **libclang engine** addresses this. When an
+:ref:`analyse.preprocessor <preprocessor_config>` table is configured and ``comment_type``
+is ``"cpp"``, each file is parsed as a real translation unit and **comments inside inactive
+preprocessor branches are dropped**. Active needs keep their original line numbers — no
+source transformation is performed.
+
+.. important:: The libclang engine requires an optional dependency:
+   ``pip install 'sphinx-codelinks[libclang]'``. The wheel bundles the native library, so
+   no compiler is required on the user's machine.
+
+How files are selected and parsed
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+File **discovery** is unchanged — :ref:`SourceDiscover <discover>` still decides which
+files are candidates. A ``compile_commands.json`` database only determines *how* each
+discovered file is parsed:
+
+.. list-table::
+   :header-rows: 1
+   :widths: 45 55
+
+   * - File
+     - How it is parsed
+   * - Listed in ``compile_commands.json``
+     - Parsed with the exact flags the compiler used for that translation unit.
+   * - A compiled source (``.c``, ``.cpp``, ``.cc``, ``.cxx``) **not** listed
+     - Skipped — assumed to be excluded from the build (e.g. another platform).
+   * - A **header** (``.h``, ``.hpp``, …) — never listed in a database
+     - Parsed **standalone** using ``defines`` and ``includes`` (see below).
+   * - Any file, when **no** database is found
+     - Parsed with ``defines`` and ``includes``.
+
+Header files
+~~~~~~~~~~~~~
+
+A ``compile_commands.json`` only ever lists compiled translation units (``.cpp`` files);
+headers are pulled in via ``#include`` and never appear as entries. **Sphinx-CodeLinks**
+therefore parses each discovered header **standalone**, using the ``defines`` and
+``includes`` you configure. Include guards resolve correctly, and ``#ifdef`` branches are
+evaluated against your ``defines``.
+
+.. note:: Because headers are parsed standalone, they see only the global ``defines`` —
+   **not** the per-file ``-D`` flags from ``compile_commands.json``. To extract a
+   particular variant's needs from headers, mirror that variant into ``defines``. Treat
+   one analysis run as **one variant**: set ``defines`` to the variant you want, and both
+   sources and headers evaluate their conditions consistently.
+
+Example
+~~~~~~~
+
+.. code-block:: cpp
+
+   // include/feature.hpp
+   #ifndef FEATURE_HPP
+   #define FEATURE_HPP
+
+   // @Always available, IMPL_BASE, impl, [REQ_BASE]
+   void base();
+
+   #ifdef VARIANT_A
+   // @Variant A only, IMPL_VAR_A, impl, [REQ_A]
+   void variant_a();
+   #endif
+
+   #endif
+
+With ``defines = ["VARIANT_A"]`` both ``IMPL_BASE`` and ``IMPL_VAR_A`` are extracted.
+With ``defines = []`` only ``IMPL_BASE`` is extracted — the ``#ifdef VARIANT_A`` block is
+inactive, so its need is dropped. The include guard (``#ifndef FEATURE_HPP``) is always
+active when the header is parsed on its own, so ``IMPL_BASE`` is never suppressed by it.

docs/source/components/configuration.rst

@@ -570,3 +570,47 @@ Configuration for marked RST block extraction.
 
 - ``start_sequence`` (``str``) - Marker that begins an RST block
 - ``end_sequence`` (``str``) - Marker that ends an RST block
+
+.. _`preprocessor_config`:
+
+analyse.preprocessor
+^^^^^^^^^^^^^^^^^^^^^
+
+Opts in to the **preprocessor-aware C/C++ engine** (powered by libclang). When this
+table is present and ``comment_type`` is ``"cpp"``, **Sphinx-CodeLinks** parses each
+C/C++ file as a real translation unit and **drops comments that fall inside inactive
+preprocessor branches** (``#if`` / ``#ifdef`` / ``#else`` …). Without this table, the
+default tree-sitter engine is used, which extracts every comment regardless of
+preprocessor conditions.
+
+See :ref:`preprocessor_engine` for the conceptual overview, header handling, and
+limitations.
+
+**Type:** ``dict``
+**Default:** Not set (the tree-sitter engine is used)
+
+.. important:: The libclang engine requires an optional dependency. Install it with
+   ``pip install 'sphinx-codelinks[libclang]'``. The ``libclang`` wheel bundles the
+   native library, so no compiler or system toolchain is required.
+
+.. code-block:: toml
+
+   [codelinks.projects.my_project.analyse.preprocessor]
+   compile_commands = "build/compile_commands.json"
+   defines = ["VARIANT_A", "PLATFORM_LINUX"]
+   includes = ["include", "third_party/include"]
+   variant_name = "variant_a"
+
+**Configuration fields:**
+
+- ``compile_commands`` (``str``) - Path to a ``compile_commands.json`` compilation
+  database. Relative paths resolve against the TOML config file. If omitted,
+  **Sphinx-CodeLinks** walks up from each source file to find one, stopping at a
+  directory containing ``.git``, ``ubproject.toml``, or ``pyproject.toml``, or at the
+  filesystem root.
+- ``defines`` (``list[str]``) - ``-D`` macros used when a file has **no** entry in the
+  database. This is the fallback applied to every header file (headers never appear in a
+  database) and to all files when no database is found.
+- ``includes`` (``list[str]``) - ``-I`` include directories used together with
+  ``defines`` on the fallback path.
+- ``variant_name`` (``str``) - Optional label echoed into the run-level output.

pyproject.toml

@@ -34,6 +34,13 @@ dependencies = [
   "tree-sitter-json>=0.24.8",
 ]
 
+[project.optional-dependencies]
+# libclang enables the preprocessor-aware C/C++ engine (skips comments inside
+# inactive #if/#ifdef regions). Plain tree-sitter C/C++ extraction needs none of
+# this, so libclang stays optional. Named after the backend, not "cpp": the
+# engine handles C and Objective-C too, and tree-sitter already covers C++.
+libclang = ["libclang>=18"]
+
 [build-system]
 requires = ["flit_core >=3.4,<4"]
 build-backend = "flit_core.buildapi"
@@ -48,6 +55,7 @@ testing = [
   "moto ~= 5.0",
   "toml>=0.10.2",
   "furo>=2024.5.6",
+  "libclang>=18",
 ]
 docs = [
   "furo>=2024.5.6",
@@ -106,8 +114,11 @@ force-sort-within-sections = true
 "**/tests/*" = [
   "ARG001",  # unused-function-argument - fixtures
   "ARG005",  # unused-lambda-argument - monkeypatches
+  "E402",    # module-import-not-at-top - pytest.importorskip guard pattern
+  "PLC0415", # import-outside-top-level - inline imports in test bodies
   "PLR2004", # magic-value-comparison - valueable for tests
   "S101",    # assert - needed for tests
+  "SIM300",  # yoda-conditions - assert order in tests is intentional
 ]
 "src/sphinx_codelinks/sphinx_extension/debug.py" = [
   "T201",  # print - used for output
@@ -140,7 +151,8 @@ plugins = ["pydantic.mypy"]
 mypy_path = "typings"
 
 [[tool.mypy.overrides]]
-module = ["licensing.*", "tomlkit.*"]
+# clang.cindex ships no type stubs / py.typed; the libclang engine wraps it.
+module = ["licensing.*", "tomlkit.*", "clang.*"]
 ignore_missing_imports = true
 
 [[tool.mypy.overrides]]

src/sphinx_codelinks/analyse/analyse.py

@@ -2,7 +2,7 @@
 from dataclasses import dataclass
 import json
 from pathlib import Path
-from typing import Any, TypedDict
+from typing import Any, TypedDict, cast
 
 from tree_sitter import Node as TreeSitterNode
 
@@ -78,9 +78,7 @@ def __init__(
         self.git_commit_rev: str | None = (
             utils.get_current_rev(self.git_root) if self.git_root else None
         )
-        self.project_path: Path = (
-            self.git_root if self.git_root else self.analyse_config.src_dir
-        )
+        self.project_path: Path = self.git_root or self.analyse_config.src_dir
         self.oneline_warnings: list[AnalyseWarning] = []
 
     def get_src_strings(self) -> Generator[tuple[Path, bytes], Any, None]:  # type: ignore[explicit-any]
@@ -112,6 +110,80 @@ def create_src_objects(self) -> None:
             self.src_files.append(src_file)
             self.src_comments.extend(src_comments)
 
+    def _resolve_preproc_args(self, src_path: Path) -> list[str] | None:
+        from sphinx_codelinks.analyse.preproc import compile_db  # noqa: PLC0415
+
+        preproc = self.analyse_config.preprocessor
+        if preproc is None:
+            return []
+        db_path = preproc.compile_commands
+        if db_path is None:
+            db_path = compile_db.find_compile_db(src_path, self.project_path)
+        if db_path is not None and db_path.is_file():
+            flags = compile_db.load_flags_map(db_path)
+            args = flags.get(src_path.absolute().resolve())
+            if args is not None:
+                return args
+            # Absent from the DB. compile_commands.json lists only compiled
+            # translation units, never headers — so a header here is parsed
+            # standalone with the global defines (one run = one variant). A
+            # compiled source absent from the build is skipped (spec §3.3).
+            if compile_db.is_translation_unit_source(src_path):
+                return None
+            return compile_db.defines_to_args(preproc.defines, preproc.includes)
+        # No DB found at all — fall back to manual defines applied globally.
+        return compile_db.defines_to_args(preproc.defines, preproc.includes)
+
+    def create_src_objects_libclang(self) -> None:
+        from sphinx_codelinks.analyse.preproc import (  # noqa: PLC0415
+            libclang_parser,
+            loader,
+        )
+
+        # Resolve the exception via the loader (never a direct ``import
+        # clang.cindex``) so a missing ``libclang`` extra still surfaces the
+        # loader's friendly install hint rather than a bare ImportError.
+        translation_unit_load_error = (
+            loader.load_clang_cindex().TranslationUnitLoadError
+        )
+
+        for src_path in self.analyse_config.src_files:
+            if not utils.is_text_file(src_path):
+                continue
+            args = self._resolve_preproc_args(src_path)
+            if args is None:
+                logger.debug(
+                    f"codelinks: skipping {src_path} — not found in compile_commands.json"
+                )
+                continue
+            try:
+                comments = libclang_parser.extract_active_comments(src_path, args)
+            except translation_unit_load_error:
+                # Last-resort guard. Standalone parses pin ``-x`` to match the
+                # ``-std`` (see defines_to_args), so the common case — a ``.h``/
+                # ``.c`` header handed a C++ ``-std`` — now parses as C++ and its
+                # markers extract. This only fires if libclang still cannot load
+                # the file as a translation unit at all; skip it rather than
+                # aborting the whole Sphinx build. Logged at ``info`` (not
+                # ``warning``): under ``sphinx-build -W`` a warning would re-fail
+                # the very builds this guard keeps green.
+                logger.info(
+                    f"codelinks: skipping {src_path} — libclang could not load it "
+                    f"as a translation unit"
+                )
+                continue
+            if not comments:
+                continue
+            # ``c`` is a LibclangComment duck-typing the tree-sitter Node
+            # interface SourceComment reads (``.text`` / ``.start_point.row``);
+            # the Node-only path (find_associated_scope) is guarded by
+            # ``is_libclang`` so it never runs on these.
+            src_comments = [SourceComment(cast("TreeSitterNode", c)) for c in comments]
+            src_file = SourceFile(src_path.absolute())
+            src_file.add_comments(src_comments)
+            self.src_files.append(src_file)
+            self.src_comments.extend(src_comments)
+
     def extract_marker(
         self,
         text: str,
@@ -327,9 +399,12 @@ def extract_marked_content(self) -> None:
             )
             if not filepath:
                 continue
-            tagged_scope: TreeSitterNode | None = utils.find_associated_scope(
-                src_comment.node, self.analyse_config.comment_type
-            )
+            if getattr(src_comment.node, "is_libclang", False):
+                tagged_scope: TreeSitterNode | None = None
+            else:
+                tagged_scope = utils.find_associated_scope(
+                    src_comment.node, self.analyse_config.comment_type
+                )
             if self.analyse_config.get_need_id_refs:
                 anchors = self.extract_anchors(
                     text, filepath, tagged_scope, src_comment
@@ -372,7 +447,15 @@ def dump_marked_content(self, outdir: Path) -> None:
             json.dump(to_dump, f)
 
     def run(self) -> None:
-        self.create_src_objects()
+        from sphinx_codelinks.source_discover.config import CommentType  # noqa: PLC0415
+
+        if (
+            self.analyse_config.preprocessor is not None
+            and self.analyse_config.comment_type == CommentType.cpp
+        ):
+            self.create_src_objects_libclang()
+        else:
+            self.create_src_objects()
         self.extract_marked_content()
         self.merge_marked_content()
         self._log_summary()

src/sphinx_codelinks/analyse/preproc/__init__.py

@@ -0,0 +1,5 @@
+"""libclang-based, preprocessor-aware extraction engine (opt-in)."""
+
+from sphinx_codelinks.analyse.preproc import compile_db, libclang_parser, loader
+
+__all__ = ["compile_db", "libclang_parser", "loader"]