test: add extension PipelinePolicy tests

[silvi-t]

Important

This PR modifies shared/core testsuite code that could potentially affect multiple test areas. 2 reviewers should review this PR to ensure adequate coverage.

Description

• Add comprehensive test coverage for the PipelinePolicy extension (57 tests across 14 test files)
• Add PipelinePolicy client implementation for creating and managing PipelinePolicy CRDs
• Add pipeline_policy_extension_service image configuration for the ThreatAssessmentService gRPC backend
• Mark known failing tests with xfail linked to upstream issues for policy isolation, deletion reconciliation, top-level fail action, deny with CEL body, and auth+deny timeout

Changes

New: PipelinePolicy Client

• testsuite/kuadrant/extensions/pipeline_policy.py — PipelinePolicy class with methods for deny, fail, response headers, gRPC method actions, and action method definitions

New: Test Infrastructure

• conftest.py — shared fixtures: pipeline_policy, threat_assessment_service, CRD existence check (check_pipeline_policy_crd)
• interactions/conftest.py — fixtures for AuthPolicy/RateLimitPolicy interaction tests: authorization, auth, rate_limit
• config/settings.yaml / config/settings.local.yaml.tpl — added pipeline_policy_extension_service.image setting
• testsuite/utils/constants.py — added THREAT_ASSESSMENT_THRESHOLD and EXTENSION_POLICY_PROPAGATION_WAIT

Known Issues (xfail)

• kuadrant-operator#2023 — policy actions leak to untargeted routes and gateways
• kuadrant-operator#2022 — targeting non-existent resources reports Accepted: True
• kuadrant-operator#2009 — EnvoyFilter not reconciled after PipelinePolicy deletion
• kuadrant-operator#2018 — withBody should be a CEL expression
• kuadrant-operator#2015 — top-level fail action should be rejected
• wasm-shim#371 — deny action after auth gRPC call causes 30s timeout

Verification steps

Run all PipelinePolicy tests sequentially:

poetry run pytest -vv testsuite/tests/singlecluster/extensions/pipeline_policy/

[!NOTE]
Due to the policy isolation bug ([kuadrant-operator#2023](https://github.com/Kuadrant/kuadrant-operator/issues/2023)), running tests in parallel with `-n4` may cause additional test failures. PipelinePolicy actions leak
across routes and gateways, so policies from parallel test modules can interfere with each other. Sequential execution is recommended until the isolation bug is fixed.

Closes #974, #975, #976

Summary by CodeRabbit

Release Notes

• New Features

  • Added PipelinePolicy extension support enabling request and response pipeline actions including deny rules, fail conditions, header manipulation, and gRPC integration for threat assessment.
  • Added configuration support for threat assessment service integration.

• Tests

  • Added comprehensive test suite covering PipelinePolicy functionality: basic operations, action composition, AuthPolicy and RateLimitPolicy interactions, lifecycle management, error handling, and isolation verification.

[coderabbitai]

📝 Walkthrough

Walkthrough

This PR introduces comprehensive end-to-end testing for PipelinePolicy, an out-of-tree Kuadrant extension. It adds a Python model for PipelinePolicy with request/response action builders, shared test fixtures including threat assessment service deployment, and test modules covering basic routing, action types (deny, fail, gRPC, headers), targeting scopes, lifecycle operations, validation, and cross-policy interactions.

Changes

PipelinePolicy Extension Testing Infrastructure

Layer / File(s)	Summary
Configuration and build infrastructure Makefile, config/settings.local.yaml.tpl, config/settings.yaml, testsuite/utils/constants.py	Adds pipelinepolicies.extensions.kuadrant.io to Makefile clean target, introduces pipeline_policy_extension_service configuration with threat assessment service image, and centralizes test thresholds (THREAT_ASSESSMENT_THRESHOLD, EXTENSION_POLICY_PROPAGATION_WAIT) as constants.
PipelinePolicy model and base test fixtures testsuite/kuadrant/extensions/pipeline_policy.py, testsuite/tests/singlecluster/extensions/pipeline_policy/conftest.py	Introduces ActionSection class that wraps pipeline action lists with methods to append gRPC methods, deny rules, fail predicates, and response headers; PipelinePolicy class extending Policy with cached on_http_request and on_http_response section accessors; shared pytest fixtures for CRD validation, instance creation, threat assessment service deployment, and resource lifecycle management.
Basic happy-path and composition semantic tests testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_basic.py, test_pipeline_policy_composition.py	Validates basic request/response routing with custom headers, action ordering and short-circuit behaviour (deny winning over fail), multiple deny predicates as OR logic, empty/unidirectional pipelines, and gRPC response variable usage in subsequent actions.
Deny action tests testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_deny.py	Tests path-based and header-based deny predicates, custom denial status codes and response headers, CEL-based custom response bodies, multiple deny rules as OR logic, and response-phase denial with backend status override.
gRPC action tests testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc.py, test_pipeline_policy_grpc_errors.py, test_pipeline_policy_fail.py	Validates gRPC upstream method configuration and invocation, conditional execution via request header predicates, threat-level-based decision logic, error handling for unavailable upstreams and invalid service/method references, and fail-action short-circuit triggered by gRPC response variables.
Response-action and targeting tests testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_response.py, test_pipeline_policy_targeting.py, test_pipeline_policy_isolation.py	Tests response-phase header injection with conditional predicates, gateway-level targeting across multiple HTTPRoutes, and HTTPRoute/gateway-level isolation to ensure policies affect only their targets.
Lifecycle management and validation tests testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_lifecycle.py, test_pipeline_policy_validation.py	Validates resource update and deletion propagation, invalid target detection (HTTPRoute/Gateway not found), malformed CEL predicates, gRPC variable forward-references, and duplicate gRPC variable names.
Cross-policy interaction tests testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/conftest.py, test_pipeline_policy_auth.py, test_pipeline_policy_ratelimit.py, test_pipeline_policy_auth_ratelimit.py	Tests PipelinePolicy interactions with AuthPolicy (OIDC authentication, denial precedence), RateLimitPolicy (combined header and rate-limit enforcement), and all three policies together; includes fixtures for OIDC identity verification and low-threshold rate limits.

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~60 minutes

Possibly related issues

• PipelinePolicy model, fixtures, and basic happy-path test #975 — Directly implements the PipelinePolicy model, fixtures, and comprehensive happy-path tests specified in the epic sub-issue.
• Comprehensive PipelinePolicy test cases #976 — Directly implements comprehensive PipelinePolicy test cases covering all action types, edge cases, error paths, and cross-policy interactions.

Possibly related PRs

• Kuadrant/testsuite#980 — Extends the same testsuite/utils/constants.py module with additional test constants and thresholds for extension policy testing.

Suggested labels

Test case

Suggested reviewers

• trepel
• averevki
• crstrn13
• azgabur

---

🐰 A new extension hops into view, ✨
PipelinePolicy tests now run without fear, 🚀
Fixtures built, assertions clear, 💚
gRPC flows and denials sincere, 🎯
Kuadrant SDK shines this year! 🌟

🚥 Pre-merge checks | ✅ 5

✅ Passed checks (5 passed)

Check name	Status	Explanation
Title check	✅ Passed	The title 'test: add extension PipelinePolicy tests' accurately and concisely describes the primary change—adding comprehensive test coverage for the PipelinePolicy extension.
Description check	✅ Passed	The PR description comprehensively covers objectives, changes, known issues with issue tracking, and verification steps—meeting all template requirements.
Linked Issues check	✅ Passed	All linked issue requirements are met: PipelinePolicy client implementation, comprehensive test coverage across 57 tests, fixtures for resource management, configuration settings, and known issues marked with xfail.
Out of Scope Changes check	✅ Passed	All changes are strictly scoped to testing infrastructure: PipelinePolicy client library, test cases, fixtures, configuration, and constants—no deployment or unrelated modifications.
Docstring Coverage	✅ Passed	Docstring coverage is 98.06% which is sufficient. The required threshold is 80.00%.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[coderabbitai]

🧹 Nitpick comments (3)

testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py (1)

69-73: TODO: assert the expected validation message once available.

Until the message is asserted, this test only checks Accepted=False, which could pass for an unrelated rejection reason. Tighten it when the upstream validation lands.

Would you like me to open a tracking issue for adding the message assertion here?

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py`
around lines 69 - 73, Test currently only asserts
policy.wait_until(has_condition("Accepted", "False"), ...) which can match
unrelated rejections; update the test to also assert the expected validation
message once the upstream validation is implemented by replacing the TODO with
an assertion that the policy's condition contains the exact expected message
(use policy.refresh().model.status.conditions to read conditions) — for example,
extend has_condition usage or add an assert that any(c.type == "Accepted" and
c.status == "False" and expected_message in c.message for c in
policy.refresh().model.status.conditions) where expected_message is the upstream
validation text; leave a TODO/issue reference if the message is not yet
available.

testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py (1)

8-22: ⚖️ Poor tradeoff

Optional: consider centralising the duplicated pipeline_policy/commit fixtures.

The header-only pipeline_policy fixture and the commit autouse fixture are repeated (with small variations in committed components) across the three interaction modules. A shared parametrised helper in interactions/conftest.py could reduce drift, though the per-module differences make this a low priority.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py`
around lines 8 - 22, Extract the duplicated pipeline_policy fixture and the
autouse commit fixture into a shared interactions/conftest.py: move the
header-only pipeline_policy setup (the pipeline_policy fixture that calls
pipeline_policy.on_http_response.add_headers([...]) and returns pipeline_policy)
and the commit fixture (which iterates components, calls
request.addfinalizer(component.delete), component.commit(), and
component.wait_for_ready()) into conftest.py, and make commit parametrisable so
callers can pass which components to commit (e.g., via pytest param or an
injected fixture) while preserving autouse behavior where needed; update the
three interaction test modules to import/rely on the shared pipeline_policy and
to call the commit fixture with the appropriate component list to avoid
duplication.

testsuite/kuadrant/extensions/pipeline_policy.py (1)

77-83: 💤 Low value

Inconsistent header serialisation between add_headers and add_deny.

add_headers serialises the headers list via str(headers), producing a Python repr with single quotes (e.g. [['x-single', 'one']]), whereas add_deny(with_headers=...) expects a caller-supplied double-quoted string (e.g. '[["x-deny-reason", "blocked"]]'). This only works if the extension parses headersToAdd/withHeaders as CEL (single quotes valid) rather than strict JSON. Consider accepting a List[List[str]] in both methods and serialising consistently to avoid surprises for future callers.

Please confirm the extension accepts a Python-style single-quoted list for headersToAdd (i.e. CEL evaluation rather than JSON parsing); the response-header tests should fail if it does not.

🤖 Prompt for AI Agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@testsuite/kuadrant/extensions/pipeline_policy.py` around lines 77 - 83, The
add_headers method currently serialises headers with Python's str(headers)
producing single quotes, causing inconsistency with add_deny's expected
double-quoted JSON; change add_headers (and ensure add_deny) to accept headers
as a List[List[str]] and serialise them with json.dumps before placing into the
action dict (keys "headersToAdd" and "withHeaders"), preserving predicate
handling and the `@modify` decorator so both methods produce consistent JSON-style
double-quoted strings for downstream parsing.

🤖 Prompt for all review comments with AI agents

Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Nitpick comments:
In `@testsuite/kuadrant/extensions/pipeline_policy.py`:
- Around line 77-83: The add_headers method currently serialises headers with
Python's str(headers) producing single quotes, causing inconsistency with
add_deny's expected double-quoted JSON; change add_headers (and ensure add_deny)
to accept headers as a List[List[str]] and serialise them with json.dumps before
placing into the action dict (keys "headersToAdd" and "withHeaders"), preserving
predicate handling and the `@modify` decorator so both methods produce consistent
JSON-style double-quoted strings for downstream parsing.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py`:
- Around line 8-22: Extract the duplicated pipeline_policy fixture and the
autouse commit fixture into a shared interactions/conftest.py: move the
header-only pipeline_policy setup (the pipeline_policy fixture that calls
pipeline_policy.on_http_response.add_headers([...]) and returns pipeline_policy)
and the commit fixture (which iterates components, calls
request.addfinalizer(component.delete), component.commit(), and
component.wait_for_ready()) into conftest.py, and make commit parametrisable so
callers can pass which components to commit (e.g., via pytest param or an
injected fixture) while preserving autouse behavior where needed; update the
three interaction test modules to import/rely on the shared pipeline_policy and
to call the commit fixture with the appropriate component list to avoid
duplication.

In
`@testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py`:
- Around line 69-73: Test currently only asserts
policy.wait_until(has_condition("Accepted", "False"), ...) which can match
unrelated rejections; update the test to also assert the expected validation
message once the upstream validation is implemented by replacing the TODO with
an assertion that the policy's condition contains the exact expected message
(use policy.refresh().model.status.conditions to read conditions) — for example,
extend has_condition usage or add an assert that any(c.type == "Accepted" and
c.status == "False" and expected_message in c.message for c in
policy.refresh().model.status.conditions) where expected_message is the upstream
validation text; leave a TODO/issue reference if the message is not yet
available.

---

ℹ️ Review info

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: d003d1c4-2f1a-4a81-ac35-7eac9f6ba6e9

📥 Commits

Reviewing files that changed from the base of the PR and between 4796963 and 25a4747.

📒 Files selected for processing (23)

• Makefile
• config/settings.local.yaml.tpl
• config/settings.yaml
• testsuite/kuadrant/extensions/pipeline_policy.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/__init__.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/conftest.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/__init__.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/conftest.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_auth.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_auth_ratelimit.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/interactions/test_pipeline_policy_ratelimit.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_basic.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_composition.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_deny.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_fail.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_grpc_errors.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_isolation.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_lifecycle.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_response.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_targeting.py
• testsuite/tests/singlecluster/extensions/pipeline_policy/test_pipeline_policy_validation.py
• testsuite/utils/constants.py

[crstrn13]

this one has been merged, shouldn't be failing anymore.

[crstrn13]

issue has been merged, you could remove the xfail.

[crstrn13]

issue has been merged, you could remove the xfail.

[crstrn13]

issue has been merged, you could remove the xfail.

[crstrn13]

issue has been merged, you could remove the xfail.

[crstrn13]

issue has been merged, you could remove the xfail.

[crstrn13]

this was an issue in the PipelinePolicy reconciler. I just pushed a fix that should validate if the targetRef exits or not before proceeding with comitting the pipeline.

[crstrn13]

issue has been closed. should remove the xfail.

[crstrn13]

this bit is reused in a lot of tests below. Thinking maybe we could wrap this into some sort of function, maybe the whole pipelinepolicy creation, similar to how other fixtures, such as api_key are being handels.

[crstrn13]

very cool test

[crstrn13]

This will not work, because obj.exists() returns a tuple. One way I managed to make it work

Suggested change

	assert policy.wait_until(lambda obj: not obj.exists(), timelimit=30), "PipelinePolicy was not deleted"
	assert not policy.committed, "PipelinePolicy was not deleted"