fix(executor): bound consecutive Plugin.Handle errors with a system-failure counter

[pingsutw]

Why are the changes needed?

When the k8s API rejects pod creation via an admission webhook (e.g. flyte-pod-webhook denies the request because a referenced secret cannot be found in any configured secret manager), launchResource in executor/pkg/plugin/k8s/plugin_manager.go returned the error as a raw Go error with pluginsCore.UnknownTransition and no phase transition.

In taskaction_controller.go, when Plugin.Handle returns a Go error, the controller only logs "Plugin Handle failed", emits a FailedPluginHandle event, and requeues with the default duration. It does not record a phase, increment any system-failure counter, or convert to PermanentFailure. As a result, the TaskAction stays in Queued forever — the webhook will keep denying the same request, and there is no max-system-failure handling on this path.

Repro:

@env.task
def fn(x: int) -> int:
    print("my secret is: ", os.getenv("TEST1"))   # secret "test1" does not exist
    ...

Before the fix, the TaskAction loops forever with the webhook denial message in the controller log:

admission webhook "flyte-pod-webhook.flyte.org" denied the request:
  none of the secret managers injected secret [key:"test1" ...]

What changes were proposed in this pull request?

In launchResource, before the generic system-error fall-through:

1. Detect admission-webhook denials via a small helper isAdmissionWebhookDenial(err) that matches the canonical error string ("admission webhook" + "denied the request").
2. Also detect k8serrors.IsInvalid(err) (invalid pod specs are equally non-transient).
3. Return pluginsCore.PhaseInfoFailure("AdmissionDenied", err.Error(), nil) — a permanent failure phase — so the TaskAction terminates immediately with the underlying webhook message preserved for the user.

How was this patch tested?

Manual end-to-end test using the bundled devbox:

1. Rebuilt flyte-devbox:latest with this change (make -C docker/devbox-bundled build) and restarted the devbox.
2. Ran a task that references a non-existent secret (secrets=["test1"]).
3. Confirmed the TaskAction transitions to PermanentFailure in ~8s (single attempt) instead of getting stuck:

NAMESPACE                 NAME                      RUN                    TASKTYPE   STATUS             AGE
flytesnacks-development   rf5v6h9d79n2ndnnqkpr-a0   rf5v6h9d79n2ndnnqkpr   python     PermanentFailure   8s

Error State:

Code:    AdmissionDenied
Kind:    USER
Message: admission webhook "flyte-pod-webhook.flyte.org" denied the request: ...

Labels

• fixed

Check all the applicable boxes

• I updated the documentation accordingly.
• All new and existing tests passed.
• All commits are signed-off.

• main
  • Flyte 2: Cut over to Flyte 2 new backend implementation #6583
    • fix(executor): bound consecutive Plugin.Handle errors with a system-failure counter 👈

[AdilFayyaz]

are we resetting this back to 0? Suppose we hit 2 system errors and then a success, shouldn't this be reset?

[pingsutw]

I'm not sure actually. we don't reset it in v1.

[AdilFayyaz]

wouldnt it be an issue for long running tasks that may have transient failures followed by successes in between. Task could silently die after accumulating nonconsecutive maxSystemFailures

[pingsutw]

that makes sense, let me update it

[pingsutw]

updated it