[Inference] 08 - Integrate workflow anonymization pipeline into chat_complete path - #288786
Draft
jonwalstedt wants to merge 54 commits into
Draft
[Inference] 08 - Integrate workflow anonymization pipeline into chat_complete path#288786jonwalstedt wants to merge 54 commits into
jonwalstedt wants to merge 54 commits into
Conversation
|
🤖 Jobs for this PR can be triggered through checkboxes. 🚧
ℹ️ To trigger the CI, please tick the checkbox below 👇
|
This was referenced Sep 2, 2026
[Inference / Workflows] Add synchronous workflow execution and anonymization type foundation
#280715
Closed
Open
Open
Draft
jonwalstedt
changed the title
jonwalstedt
force-pushed
the
anon/08-pipeline-integration
branch
2 times, most recently
from
September 8, 2026 10:41
1924a1f to
e03b20a
Compare
8 tasks
jonwalstedt
force-pushed
the
anon/08-pipeline-integration
branch
from
September 9, 2026 11:13
90fe103 to
c7e2371
Compare
jonwalstedt
added
backport:skip
jonwalstedt
force-pushed
the
anon/08-pipeline-integration
branch
from
September 10, 2026 10:40
2fbeeae to
51d663a
Compare
Operator sanity guard — 512 characters is generous for any real HMAC key (RFC 2104 hashes keys longer than 64 bytes anyway) but catches accidental full-file pastes or misconfigured values at startup. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…trolled thread pre-warming Replaces the implicit minThreads override (which silently forced minThreads to maxThreads when workflowDriven was enabled) with an explicit xpack.inference.workers.anonymization.workflowDrivenMinThreads config key (default 3). Operators running workflowDriven: false are unaffected. Operators running workflowDriven: true can lower the value to allow partial thread scaling, or set it to 0 to accept cold-start risk. Values above maxThreads produce a startup warning and are clamped. Also removes unsupported `sensitive: true` from encryptionKey schema.string() options and drops the unused workflowAnonymizationOptions class field. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…_workflows integration Adds the public server contract for workflow-driven anonymization: - Exports capability helpers (createPiiTokenizationCapabilityValue, createInferenceProceedCapabilityValue, resolve variants) from inference plugin server index - New inference_workflows step handlers: ai.pii, call_site.proceed, transform.pii_restore - Token map, message records, anonymization metrics, capabilities helpers - inference_workflows plugin wired to register the anonymization provider with the inference setup contract - inference_workflows kibana.jsonc gains required plugin deps Runtime no-op until the pipeline integration (PR 8) wires the around-completion hook. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ependency Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Kibana config schema properties are camelCase; snake_case in the YAML path reference was incorrect. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ete path Wire up the aroundCompletion hook's anonymization provider to the callback API: detect PII via the standalone PiiRegexWorkerService, tokenize with HMAC-SHA256 entity tokens, stream-restore content and tool-call arguments, and fall back (or block) based on the configured failureMode. Includes the streaming content restorer (with MIN_PREFIX_HOLDBACK=2 trade-off documented in tests), OTel metrics for first-chunk latency and request outcomes, and the inference_workflows integration test as a release gate. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…adata field PR 04 introduced WorkflowAnonymizationContext on ChatCompleteMetadata.workflowAnonymization to avoid coupling to ChatCompleteAnonymizationMetadata (Steph's Anonymization Platform Service, scheduled for deletion). Update the pipeline read path accordingly.
… options in pipeline PR 04 adds agentId flat on ChatCompleteMetadata (no wrapper). sessionId is already a top-level ChatCompleteOptions field available in scope. Update the pipeline read path accordingly.
…se path Mirror of the anon/06 fix: thread encryptionKey through WorkflowAnonymizationOptions and read it directly in the pipeline instead of pulling it from anonymization.saltPromise (which is always undefined while ANONYMIZATION_FEATURE_ACTIVE is hardcoded false). Removes saltPromise from the workflow pipeline interface entirely — the legacy path keeps its own saltPromise marked for deletion with the anonymization plugin. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Marks the config value as sensitive so it is redacted from diagnostics, and adds a 512-character maxLength as an operator sanity guard. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…trolled thread pre-warming Replaces the implicit minThreads override (which silently forced minThreads to maxThreads when workflowDriven was enabled) with an explicit xpack.inference.workers.anonymization.workflowDrivenMinThreads config key (default 3). Operators running workflowDriven: false are unaffected. Operators running workflowDriven: true can lower the value to allow partial thread scaling, or set it to 0 to accept cold-start risk. Values above maxThreads produce a startup warning and are clamped. Also removes unsupported `sensitive: true` from encryptionKey schema.string() options. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
sessionId is a top-level ChatCompleteOptions field; agentId is flat on ChatCompleteMetadata. The test was written against the original metadata.anonymization.* shape that was refactored away by two successive fix commits. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…RegexWorker to tests - PiiRegexWorkerService pool is now only created when workflowDriven is true, fulfilling the zero-overhead-when-off guarantee for the Piscina pool. resolveWorkflowAnonymizationOptions is called only when the worker exists. - api.test.ts: import createPiiRegexWorkerServiceMock and pass piiRegexWorker in both workflow-mode workflowAnonymization objects. - plugin.test.ts: add piiRegexWorker to all resolveWorkflowAnonymizationOptions call sites and include it in the expected return values where enabled: true. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… and workflowAnonymization blocks Follows the same split as 06-inference-anon-config. Each service now has independent config; enabling/disabling one pool does not affect the other. PiiRegexWorkerService uses WorkflowAnonymizationWorkerConfig from workers.workflowAnonymization; RegexWorkerService keeps workers.anonymization. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
In Zod v4 classic, `.unwrap()` and `.removeDefault()` return `core.$ZodType` (the raw base type), not `z.ZodType` (the extended classic layer). The reassignment `value = value.unwrap()` therefore fails the type checker. Fix by casting to `z.ZodType` (accurate at runtime — all Kibana schemas use the classic layer) and replace the deprecated `.removeDefault()` with `.unwrap()`. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…+ add tests
withTemplateStringSupport() was accepting any string for array params via
z.string(), which silently passes plain mistyped values through the YAML editor
schema. Replace with a bounded regex that only accepts whole-value Liquid
expressions ({{ expr }} / ${{ expr }}), matching all real-world usage seen in
workflow YAML examples.
Also adds TEMPLATE_EXPRESSION_MAX_LENGTH = 500 to constants, and a regression
test suite covering: valid templates, rejected plain/partial strings, length
limit, optional and default-wrapped arrays, non-array fields unchanged, and
object unknownKeys policy preservation after .extend().
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
extend() throws for ZodObject schemas that contain object-level refinements (.refine / .superRefine). Because withTemplateStringSupport() calls extend() on the connector's paramsSchema unconditionally (when array fields exist), a single connector with a refined paramsSchema caused the entire workflow schema construction to throw, rather than just losing that one connector's widening. Fix by switching to safeExtend(), which preserves refinements and the unknownKeys policy (strict/passthrough). Also add an early return when there are no array fields to widen, avoiding any extend call when no changes are needed. Adds a regression test covering: schema construction does not throw for a refined paramsSchema, template strings still accepted, and the object-level refinement is preserved after widening. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ness
- Extract WHOLE_VALUE_TEMPLATE_EXPRESSION_REGEX + isWholeValueTemplateExpression
to common/template_expressions.ts; move TEMPLATE_EXPRESSION_MAX_LENGTH there.
Single definition shared by schema and the new cross-package test.
- Fix regex: bare {{ }} and padded ${{ }} forms can never resolve to an array
at runtime (templating_engine.ts:98 checks startsWith/endsWith without trim),
and multi-expression concatenations throw mid-execution. New regex requires
${{, forbids inner }}, and drops the \s* tolerance that diverged from runtime.
- Fix withTemplateStringSupport: .default() was unwrapped but never re-applied,
turning z.array(...).default([]) into a required field. Captured and restored.
Loop replaces if-chain so stacked .optional().default() in any order also works.
Verified against the real InferenceRerankParamsSchema shape.
- Hoist LIQUID_TEMPLATE_SCHEMA to module scope (was rebuilt per field per connector).
- Correct withTemplateStringSupport doc comment: this schema gates workflow
create/update in workflows_management server-side, not just the Monaco editor.
- Expand supportedExecutionModes JSDoc: broaden the async-only criterion to cover
scheduling work (workflow.executeAsync), document the fail-open default with
explicit reasoning and the trade-off.
- Add builtin_step_definitions.test.ts: pins all five async-only steps and the
completeness set, so a dropped annotation fails CI rather than hanging a request.
- Add template_expressions_runtime.test.ts in workflows_execution_engine: drives
each accepted template form through WorkflowTemplatingEngine and asserts the
result is an array; pins rejected forms with per-case rationale. Cross-package
invariant that prevents schema and runtime from drifting.
- Update PR description: fix inverted validateSyncWorkflow sentence, correct
blast-radius note, add what the new tests establish.
Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
The test claimed to verify the advance-one-char guard for zero-length matches, but a+ can never produce a zero-length match so the guard was never exercised. Switching to a* causes RE2 to emit zero-length hits between non-'a' characters; the guard now fires and both 'aaa' runs are still found, as asserted. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…rkerService Adds RE2JS.compile() guard in runSync so patterns requiring native RegExp (lookahead/lookbehind/backreferences) throw before execution when the worker pool is disabled. The worker pool provides ReDoS containment via isolation and task timeout; the sync path has neither, so catastrophic backtracking on the event loop must be prevented at the gate. Adds test asserting the sync path rejects a positive-lookahead pattern. Renames AnonymizationWorkerConfig to WorkflowAnonymizationWorkerConfig to reflect the dedicated pool; the placeholder alias in config.ts is superseded when merged with 06-inference-anon-config. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
… rebuild Address review feedback: on task timeout, destroying and recreating the whole Piscina pool rejects all other in-flight detection tasks sharing the pool (up to maxThreads concurrently), not just the timed-out one. Piscina already terminates the specific worker thread when the task's AbortSignal fires, so relying on that signal alone contains the runaway task without collateral impact to sibling requests. Reviewed-at: elastic#288758 (comment) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- entity_mask.test.ts: compare hash portions only in delimiter-collision test; full-token comparison always passes when entityClass prefixes differ - types.ts: correct allow_unsafe doc to reflect whole-payload fail-open behavior (returns [] for entire payload, not per-rule skipping) Finding 1 (executeRegexRules exported) was already resolved before review. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
- Remove executeRegexRules from index.ts — it is a worker-internal function, not a public API; exporting it invited callers to bypass the pool, failureMode, and the RE2 sync gate - Consolidate sync-path RE2 enforcement: add re2Only flag to compileRule and executeRegexRules so runSync no longer pre-compiles each pattern twice; a single pass now both validates and compiles - Drop typeof value !== 'string' guard in executeRegexRules — Record<string,string> guarantees string values by type; the check was contradicting the contract - Add distinct error message for Piscina queue-at-capacity errors so operators can tell queue saturation apart from bad-pattern failures - Validate non-empty entityClass in generateEntityToken - Use WorkflowAnonymizationWorkerConfig in test (was AnonymizationWorkerConfig, will diverge when elastic#288762 lands) - Use service.stop() in afterEach instead of casting to any - Add assert_re2_compilable.test.ts covering valid patterns, RE2-unsupported constructs, invalid syntax, and error message content - Add maxQueue overflow tests using a spy on the internal Piscina run method - Update config.ts placeholder comment with TODO(elastic#288762) Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ce.run() Address review feedback: the JSDoc said "logs and skips the offending rule" but the implementation catches any payload failure and returns [] for the entire payload, not per-rule. Updated to "logs and returns no matches for the entire payload". Reviewed-at: elastic#288758 (comment)
…RegexWorkerService Address review feedback: the service matched 'Task queue is at capacity' but Piscina 5.3.1 throws 'Task queue is at limit', so the saturation branch never fired. Fixed the string in the service and the mock in the test. Piscina does not expose a stable error code, so message matching is the only option; added a comment pinning the verified version. Reviewed-at: elastic#288758 (comment)
…es.test.ts Address review feedback: the describe block was titled "skips empty and non-string field values" but the payload type is Record<string, string> so non-string values are impossible, and the test only covers empty strings. Renamed to "skips empty string fields". Reviewed-at: elastic#288758 (comment)
…RegexWorkerService Previously allow_unsafe returned [] for the entire payload when any rule failed to compile — so one broken rule silenced all other rules' matches, letting all PII through unmasked. Now invalid rules are filtered and logged individually before dispatch; surviving rules still execute and return their matches. Infrastructure failures (timeout, queue saturation, worker crash) still produce a whole-payload [] because there are no partial results to save. compileRule is exported from execute_regex_rules.ts so the pre-filter reuses the same RE2-first → native-fallback logic without duplication. On the sync path re2Only=true is forwarded, so non-RE2 patterns are also skipped (they would block the event loop). Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…d allow_unsafe semantics Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
jonwalstedt
force-pushed
the
anon/08-pipeline-integration
branch
from
September 11, 2026 09:02
5f0c17b to
cd63215
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
What
Wires the workflow anonymization provider into the
chatCompletearound-completion hook. Whenxpack.inference.anonymization.workflowDriven: trueand a matching workflow is enabled, PII is detected and tokenized before the LLM connector is called, then tokens are restored in the response. The legacy anonymization path is theelse-branch — unchanged and reachable by settingworkflowDriven: false.Also adds
anonymization_workflow.integration.test.tsas the automated release gate.Why
All the pieces are in place after PRs 1–7: the detection runtime, sync execution, step handlers, and provider registration. This PR connects them in
callback_api.tswith a singleif (workflowAnonymization)branch. Until this PR merges, no LLM call is affected regardless of configuration.The integration test is the release gate. It drives the real execution engine with real step handlers (no mocks) and asserts:
Manual checks cannot substitute for this test — it's the only automated assertion that the full pipeline works end-to-end.
Standalone boundary maintained: The new detection-context files live in
inference/server/workflow_anonymization/(alongside PR #288758's runtime), not inchat_complete/anonymization/. No new file imports o11y'sRegexWorkerService.How it fits in the stack
Wave 1, depends on #288780. After this PR merges the feature is complete but not yet enabled — PR #288787 installs the managed workflow that actually triggers anonymization. This is the last mandatory step on the critical path before the feature can be turned on per-space.
Testing
Tip
Testing the full flow: A non-merge umbrella branch combining all changes in this stack is available at #289950. Check out
anon/umbrellato test the complete end-to-end anonymization pipeline without checking out individual PRs.Stack status