Skip to content

[FEATURE] Revenue Recovery: A/B testing framework to compare retry algorithms (Smart vs Cascading) on live traffic #13324

Description

@rohanm13

Feature Description

Revenue Recovery routes every invoice through a single retry algorithm, chosen per business
profile via revenue_recovery_retry_algorithm_type (smart or cascading). Today a merchant
has to pick one - there is no way to run both on live traffic and measure which actually
recovers more revenue.

In my use-case, we want to validate that the Smart (decider/ML-based) algorithm outperforms the
existing Cascading (rules-based) one before rolling it out broadly, using real traffic rather
than offline replay.

This requests an A/B testing framework for Revenue Recovery retry routing:

  • Split live recovery traffic between the two algorithms, with a configurable ratio.
  • Deterministic, sticky assignment: an invoice is assigned to exactly one arm and must stay
    on it for its entire recovery life. An in-flight invoice must never switch arms mid-retry,
    even if the split or experiment config changes.
  • Durable record of the assignment, so recovery outcomes can be compared per variant offline.
  • No behaviour change when no experiment is active — routing falls back to the existing
    profile-configured algorithm.
  • Config/split changeable without a deploy, so an experiment can be started, ramped, or
    stopped at runtime.

Possible Implementation

Use Superposition's experimentation engine (already integrated in hyperswitch) to make the
routing decision, keyed on the invoice id, and persist the assignment on the payment intent.

1. Superposition config key revenue_recovery_routing, resolved through the existing
dimension_config! macro:

  • output: RecoveryRoutingDecision { enabled, experiment_name, variant, algorithm }
  • targeting key: GlobalPaymentId (the invoice id) → deterministic bucketing per invoice
  • dimensions: processor_merchant_id + connector
  • default value { enabled: false } → legacy routing when no experiment is running

2. Embed the arm in the value. Each experiment variant overrides the whole config value with
its own variant + algorithm, so a single resolve returns the assigned arm — no separate
applicable_variants call is needed.

3. New module core/revenue_recovery/routing.rs mapping the resolved decision to an
assignment:

  • AbRoutingResolution::{ Reused, Created, DisabledOrUnavailable, InvalidConfig }
  • disabled/unparseable → legacy routing (fail-safe)

4. Assignment chokepoint in perform_calculate_workflow:

  • if the invoice already has an assignment → reuse it (sticky; deliberately does not re-query
    Superposition)
  • else resolve from Superposition and persist
  • the assigned algorithm then overrides retry_algorithm_type for that invoice

5. Persist to the payment intent's feature_metadata (no new table):

  • path: feature_metadata.payment_revenue_recovery_metadata.recovery_routing
  • shape: RevenueRecoveryRoutingData { experiment_name, variant, assigned_algorithm, assigned_at }
  • needs a recovery_routing field on PaymentRevenueRecoveryMetadata (api_models +
    diesel_models) plus convertor passthrough
  • get_updated_feature_metadata must preserve this field, otherwise each new failed-payment
    attempt rebuilds the recovery metadata and wipes the assignment (breaking stickiness)

6. TargetingKey impl for GlobalPaymentId — must live in common_utils due to the orphan
rule.

7. Do not store the outcome. Whether an invoice recovered is already authoritative on
payment_intent.status (and amount/timing on the intent). Analytics derives per-variant recovery
by joining those with the persisted assigned_algorithm/variant, e.g.:

select recovery_routing->>'variant', count(*),
       count(*) filter (where status = 'succeeded')
from payment_intent ... group by 1;

This avoids storing redundant data and avoids writing to an intent in a terminal (succeeded)
state, which the payments update operation rejects by design.

Have you spent some time checking if this feature request has been raised before?

  • I checked and didn't find a similar issue

Have you read the Contributing Guidelines?

Are you willing to submit a PR?

Yes, I am willing to submit a PR!

Metadata

Metadata

Assignees

No one assigned

    Labels

    C-featureCategory: Feature request or enhancementS-awaiting-triageStatus: New issues that have not been assessed yet

    Type

    No type

    Fields

    No fields configured for issues without a type.

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions