Add hier config nested compliance rule feature

README.md

@@ -63,7 +63,7 @@ This App is installed in the Nautobot Community Sandbox found over at [demo.naut
 
 Full web-based HTML documentation for this app can be found over on the [Nautobot Docs](https://docs.nautobot.com/projects/golden-config/en/latest/) website:
 
-- [User Guide](https://docs.nautobot.com/projects/golden-config/en/latest/user/app_overview/) - Overview, Using the App, Getting Started, Navigating compliance (cli, json, custom), backup, app usage, intended state creation.
+- [User Guide](https://docs.nautobot.com/projects/golden-config/en/latest/user/app_overview/) - Overview, Using the App, Getting Started, Navigating compliance (cli, json, xml, custom, filtered), backup, app usage, intended state creation.
 - [Administrator Guide](https://docs.nautobot.com/projects/golden-config/en/latest/admin/install/) - How to Install, Configure, Upgrade, or Uninstall the App.
 - [Developer Guide](https://docs.nautobot.com/projects/golden-config/en/latest/dev/contributing/) - Extending the App, Code Reference, Contribution Guide.
 - [Release Notes / Changelog](https://docs.nautobot.com/projects/golden-config/en/latest/admin/release_notes/)

changes/979.added

@@ -0,0 +1 @@
+Added hier config compatability to compliance rules for matching nested config using hier config tags

docs/user/app_feature_compliance.md

@@ -63,6 +63,9 @@ For JSON based configs, the match is based on JSON's structure top level key nam
 !!! note
     "Config to Match" is mandatory for CLI configurations. If config to match is not defined for JSON, the complete JSON configuration will be compared. If the config to match is defined, comparison will take place only for defined keys.
 
+!!! tip "Advanced CLI Compliance: Hierarchical Configuration"
+    For more sophisticated CLI compliance checking, you can use **Filtered Configuration Compliance** which leverages the `hier_config`. Start the "Config to Match" field with `# hier_config` followed by YAML rule blocks using the unified `match_rules` syntax. See [Filtered Configuration Compliance](./app_feature_compliancefiltered.md) for detailed documentation and examples.
+
 !!! note
     If the data is accidentally "corrupted" with a bad tested match, simply delete the devices an re-run the compliance process.
 

docs/user/app_feature_compliancecli.md

@@ -3,6 +3,9 @@
 !!! note
     This document provides instructions for `CLI` configuration type based compliance.  The other option is `JSON` based [structured data compliance](./app_feature_compliancejson.md).
 
+!!! tip "Advanced CLI Compliance"
+    For more sophisticated CLI compliance checking with advanced filtering capabilities, see [Filtered Configuration Compliance](./app_feature_compliancefiltered.md) which provides advanced configuration matching using the `hier_config` library.
+
 ## Configuration Compliance Parsing Engine
 
 Configuration compliance is different than a simple UNIX diff. While the UI provides both, the compliance metrics are not influenced by the UNIX diff

docs/user/app_feature_compliancefiltered.md

@@ -0,0 +1,208 @@
+# Filtered Configuration Compliance
+
+## Overview
+
+Filtered Configuration Compliance is a powerful extension to the standard CLI-based compliance checking that leverages the `hier_config` library to provide advanced configuration parsing and comparison capabilities. This feature allows you to define complex matching rules to focus compliance checks on specific configuration sections or patterns.
+
+Unlike standard CLI compliance which matches configuration sections based on simple line-starting patterns, Filtered Configuration Compliance identifies and compares configuration elements based on their hierarchical relationships and specific attributes, which allows for filtering based on nested config lines.
+
+## When to Use Filtered Configuration Compliance
+
+This feature is particularly useful for:
+
+- Configurations that are benign or non-consequential to overall device operation.
+- When you are building out your compliance journey, and not prepared to include all configurations based on simple line matching.
+
+!!! warning
+    Should not be used to provide full line matches or matches that are better served as data, as an example you should not do this `- startswith: description USER PORT` but should do this `- startswith: description`.
+
+## Requirements
+
+### Platform Support
+
+Filtered configuration compliance requires:
+
+1. **CLI Configuration Type**: Only [`CLI`](./app_feature_compliancecli.md) configuration types are supported for hierarchical compliance
+2. **Platform Compatibility**: Your device platform must be supported by the `hier_config` library
+3. **Network Driver Mapping**: The platform must have a valid `hier_config` mapping in its `network_driver_mappings`
+
+## Configuring Filtered Compliance Rules
+
+### Rule Identification
+
+To create a filtered compliance rule, start the **Config to Match** field with the comment:
+
+`# hier_config`
+
+This comment must be the first line. The content that follows should be YAML consisting of one or more rule blocks. Each rule block uses the `match_rules` key containing an ordered list of predicates (e.g. `startswith`, `contains`, `re_search`, `endswith`).
+
+### Syntax
+
+Example with two rule blocks:
+
+```yaml
+# hier_config
+- match_rules:
+  - startswith: router bgp
+  - startswith: neighbor
+- match_rules:
+  - startswith: interface
+  - contains: GigabitEthernet
+```
+
+### Example Configuration Rules
+
+#### Example 1: BGP Neighbor Configuration
+
+To check compliance for all BGP neighbor configurations:
+
+```yaml
+# hier_config
+- match_rules:
+  - startswith: router bgp
+  - startswith: neighbor
+```
+
+This rule will match configurations like:
+```
+router bgp 65001
+ neighbor 10.1.1.1
+  remote-as 65002
+  description PEER_ROUTER_A
+ neighbor 10.1.1.2
+  remote-as 65003
+  description PEER_ROUTER_B
+```
+
+#### Example 2: Interface MTU Settings
+
+To check compliance for MTU settings on all interfaces:
+
+```yaml
+# hier_config
+- match_rules:
+  - startswith: interface
+  - contains: mtu
+```
+
+This rule will match configurations like:
+```
+interface GigabitEthernet0/1
+ mtu 9000
+interface GigabitEthernet0/2
+ mtu 1500
+```
+
+#### Example 3: SNMP Configuration
+
+To check compliance for SNMP server configurations:
+
+```yaml
+# hier_config
+- match_rules:
+  - startswith: snmp-server
+```
+
+This rule will match configurations like:
+```
+snmp-server community public RO
+snmp-server community private RW
+snmp-server host 192.168.1.100 version 2c public
+```
+
+#### Example 4: Access Control Lists
+
+To check compliance for specific access control list entries:
+
+```yaml
+# hier_config
+- match_rules:
+  - startswith: ip access-list
+  - contains: SECURITY
+```
+
+This rule will match configurations like:
+```
+ip access-list extended SECURITY_IN
+ permit tcp any host 192.168.1.100 eq 443
+ deny ip any any log
+ip access-list extended SECURITY_OUT
+ permit ip 192.168.1.0 0.0.0.255 any
+ deny ip any any log
+```
+
+## Fallback Behavior for Empty Intended Configuration
+
+In some cases, the intended configuration may not contain any elements that match the compliance rules, resulting in an empty intended configuration text. To handle this scenario, the system includes a fallback mechanism:
+
+**Interface Fallback**: When the intended configuration text is empty, the system automatically looks for top-level interface configurations in the running configuration and uses them as the intended configuration baseline.
+
+This fallback behavior:
+
+1. **Triggers** when `intended_text` is empty after tag-based filtering
+2. **Filters** for lines that start with "interface"
+3. **Uses** these interface declarations as the intended configuration for comparison
+
+**Example Scenario**:
+
+- Your hierarchical rule targets specific VLAN configurations
+- The intended configuration template doesn't include those VLANs
+- The running configuration has interface declarations
+- The system uses the interface lines from running config as the baseline
+
+This ensures that remediation will not remove interfaces when intended configuration is empty for an interface.
+
+## Troubleshooting
+
+### Common Issues
+
+#### Platform Not Supported
+**Error**: `Unsupported platform for hier_config v3: <platform>`
+
+**Solution**: Ensure your device platform has a valid `hier_config` mapping in its `network_driver_mappings`. Check the platform configuration in Nautobot.
+
+#### Invalid YAML Syntax
+**Error**: `Invalid YAML in match_config: <error details>`
+
+**Solution**: Validate your YAML syntax in the **Config to Match** field. Ensure proper indentation and structure.
+
+#### Wrong Configuration Type
+**Error**: `Hier config compliance rules are only supported for CLI config types`
+
+**Solution**: Change the **Config Type** to `CLI` for hierarchical compliance rules.
+
+### Debug Tips
+
+1. **Check Platform Support**: Verify your platform's `network_driver_mappings` include `hier_config`
+2. **Review hier_config Documentation**: Consult the [hier_config documentation](https://hier-config.readthedocs.io/en/latest/tags/) for advanced tagging examples
+3. **Start Simple**: Begin with basic matching rules and gradually add complexity
+4. **Empty Intended Configuration**: If you notice interface lines appearing in compliance results when not expected, check if your intended configuration contains the elements targeted by your hierarchical rules. The system automatically falls back to using running config interface declarations when intended configuration is empty
+
+## Best Practices
+
+### Rule Design
+
+1. **Specific Matching**: Create focused rules that target specific configuration elements
+2. **Logical Grouping**: Group related configuration elements in single rules when appropriate
+3. **Clear Naming**: Use descriptive feature names that clearly indicate what the rule validates
+4. **Complete Intended Configurations**: Ensure your intended configuration templates include all elements targeted by filtered compliance rules to avoid unexpected fallback behavior where interface lines from running config are used as intended baseline
+
+## Advanced Usage
+
+### Combining Multiple Rules
+
+You can create multiple hierarchical compliance rules for the same platform to check different aspects of the configuration independently.
+
+### Integration with Configuration Management
+
+Filtered configuration compliance works seamlessly with existing configuration management workflows:
+
+- Backup processes remain unchanged
+- Intended configuration generation follows the same patterns
+- Compliance reporting provides the same interface with enhanced precision
+
+## Related Documentation
+
+- [Standard CLI Compliance](./app_feature_compliancecli.md)
+- [Configuration Compliance Overview](./app_feature_compliance.md)
+- [hier_config Documentation](https://hier-config.readthedocs.io/)

mkdocs.yml

@@ -117,6 +117,7 @@ nav:
       - Navigate Compliance JSON: "user/app_feature_compliancejson.md"
       - Navigate Compliance XML: "user/app_feature_compliancexml.md"
       - Navigate Compliance Custom: "user/app_feature_compliancecustom.md"
+      - Navigate Compliance Filtered: "user/app_feature_compliancefiltered.md"
       - Navigate Intended: "user/app_feature_intended.md"
       - Navigate SoT Agg: "user/app_feature_sotagg.md"
       - Navigate Configuration Post-Processing: "user/app_feature_config_postprocessing.md"

nautobot_golden_config/nornir_plays/config_compliance.py

@@ -2,12 +2,17 @@
 
 # pylint: disable=relative-beyond-top-level
 import difflib
+import hashlib
+import json
 import logging
 import os
 from collections import defaultdict
 from datetime import datetime
 
+import hier_config
+import yaml
 from django.utils.timezone import make_aware
+from hier_config.utils import HCONFIG_PLATFORM_V2_TO_V3_MAPPING
 from lxml import etree
 from nautobot_plugin_nornir.constants import NORNIR_SETTINGS
 from nautobot_plugin_nornir.plugins.inventory.nautobot_orm import NautobotORMInventory
@@ -16,6 +21,7 @@
 from nornir.core.plugins.inventory import InventoryPluginRegister
 from nornir.core.task import Result, Task
 from nornir_nautobot.exceptions import NornirNautobotException
+from pydantic import TypeAdapter
 
 from nautobot_golden_config.choices import ComplianceRuleConfigTypeChoice
 from nautobot_golden_config.exceptions import ComplianceFailure
@@ -171,8 +177,13 @@
     intended_cfg = _open_file_config(intended_file)
 
     for rule in rules[obj.platform.network_driver]:
-        _actual = get_config_element(rule, backup_cfg, obj, logger)
-        _intended = get_config_element(rule, intended_cfg, obj, logger)
+        # Check for hier config compliance rule
+        section_type = rule["section"][0]
+        if "# hier_config" in section_type:
+            _actual, _intended = process_nested_compliance_rule_hier_config(rule, backup_cfg, intended_cfg, obj, logger)
+        else:
+            _actual = get_config_element(rule, backup_cfg, obj, logger)
+            _intended = get_config_element(rule, intended_cfg, obj, logger)
 
         # using update_or_create() method to conveniently update actual obj or create new one.
         ConfigCompliance.objects.update_or_create(
@@ -194,6 +205,98 @@
     return Result(host=task.host)
 
 
+def process_nested_compliance_rule_hier_config(rule, backup_cfg, intended_cfg, obj, logger):  # pylint: disable=too-many-locals
+    """
+    Processes nested compliance rules using hierarchical configuration comparison.
+
+    This function compares backup (running) and intended configurations using hierarchical config parsing.
+    It applies tag-based filtering according to match criteria defined in the compliance rule's match_config.
+
+    Args:
+        rule (dict): Contains compliance rule details, including match configuration.
+        backup_cfg (str): The backup (running) configuration as a string.
+        intended_cfg (str): The intended configuration as a string.
+        obj (object): Object containing platform and device information.
+        logger (logging.Logger): Logger instance for logging errors and debug information.
+
+    Returns:
+        tuple[str, str]: Filtered running and intended configuration text for compliance comparison.
+
+    Notes:
+        The match config must have # hier_config as a comment at the top of the YAML block.
+
+    The match_config field in the rule should define hierarchical config tags and matching criteria in YAML format.
+    See: https://hier-config.readthedocs.io/en/latest/tags/
+    """
+    # Check if the compliance rule is of type CLI
+    if rule["obj"].config_type != ComplianceRuleConfigTypeChoice.TYPE_CLI:
+        error_msg = "Hier config compliance rules are only supported for CLI config types."
+        logger.error(error_msg, extra={"object": obj})
+        raise NornirNautobotException(error_msg)
+
+    # Convert v2 platform to v3 platform
+    v2_platform = obj.platform.network_driver_mappings.get("hier_config")
+    platform = HCONFIG_PLATFORM_V2_TO_V3_MAPPING.get(v2_platform)
+
+    if not platform:
+        error_msg = f"Unsupported platform for hier_config v3: {v2_platform}"
+        logger.error(error_msg, extra={"object": obj})
+        raise NornirNautobotException(error_msg)
+
+    # Create config objects using v3 API
+    running_config = hier_config.get_hconfig(platform_or_driver=platform, config_raw=backup_cfg)
+    generated_config = hier_config.get_hconfig(platform_or_driver=platform, config_raw=intended_cfg)
+
+    v3_tags = tuple()
+    tag_names = set()
+
+    # Create hier config v3 tags
+    try:
+        match_config = yaml.safe_load(rule["obj"].match_config)
+        for tag_rule in match_config:
+            # Create a unique tag name for each match rule
+            tag_name = hashlib.sha1(json.dumps(tag_rule, sort_keys=True).encode()).hexdigest()  # noqa: S324
+            tag_rule["apply_tags"] = frozenset([tag_name])
+            tag_names.add(tag_name)
+        v3_tags = TypeAdapter(tuple[hier_config.models.TagRule, ...]).validate_python(match_config)
+    except yaml.YAMLError as e:
+        error_msg = f"Invalid YAML in match_config: {str(e)}"
+        logger.error(error_msg, extra={"object": obj})
+        raise NornirNautobotException(error_msg) from e
+
+    # Apply tags to the running and generated configs
+    for tag_rule in v3_tags:
+        for child in running_config.get_children_deep(tag_rule.match_rules):
+            child.tags_add(tag_rule.apply_tags)
+
+    for tag_rule in v3_tags:
+        for child in generated_config.get_children_deep(tag_rule.match_rules):
+            child.tags_add(tag_rule.apply_tags)
+
+    # Filter configs by the applied tags
+    running_config_generator = running_config.all_children_sorted_by_tags(tag_names, set())
+    running_text = "\n".join(c.cisco_style_text() for c in running_config_generator)
+
+    intended_config_generator = generated_config.all_children_sorted_by_tags(tag_names, set())
+    intended_text = "\n".join(c.cisco_style_text() for c in intended_config_generator)
+
+    # If the intended text is None, inject the top level parent lines from running config into the intended config.
+    # This will prevent the removal of top-level interfaces or other sections that are not matched by the tags
+    # For example, if the intent is to remove 'flow exporter' from the config when remediating
+    # Instead of this:
+    # no interface GigabitEthernet0/0
+    # we will have this:
+    # no flow exporter 192.0.0.1
+    if not intended_text:
+        running_config_top_level_lines = []
+        for child in running_config.all_children_sorted_by_tags(tag_names, set()):
+            if child.real_indent_level == 0 and child.cisco_style_text().startswith("interface"):
+                running_config_top_level_lines.append(child.cisco_style_text())
+        intended_text = "\n".join(running_config_top_level_lines)
+
+    return running_text, intended_text
+
+
 def config_compliance(job):  # pylint: disable=unused-argument
     """
     Nornir play to generate configurations.