[FEATURES] Extend Cache Runtime Interface to Support Full Data Lifecycle and In-Place Operations

[cheyang]

What feature you'd like to add:

1. Extend Generic Cache Runtime Interface:

   • Support complete data lifecycle management including data loading, data processing workflows, and cache-aware data mutations
   • Add state machine support for managing data operation lifecycle and state transitions
   • Define standardized API for cache runtime operations beyond basic create/delete

2. In-Place Cache Upgrade:

   • Enable cache engine version upgrades without disrupting workloads or requiring dataset re-provisioning
   • Support rolling upgrade strategies with minimal downtime (target: < 5 minutes)
   • Implement rollback mechanisms for failed upgrades

3. In-Place Cache Rebuild:

   • Automatic cache recovery after node failures without manual intervention
   • Support configuration changes without dataset recreation
   • Maintain data availability during rebuild operations

4. Reference Adapter Implementations:

   • Curvine adapter implementing the extended interface
   • Alluxio adapter implementing the extended interface
   • Comprehensive test coverage (80%+ unit tests, E2E test scenarios)

Why is this feature needed:

The current Cache Runtime interface lacks comprehensive support for end-to-end data operations and dynamic runtime changes, forcing users to delete and recreate datasets for routine maintenance (e.g., engine upgrades or recovery from failures). This leads to unnecessary downtime, operational overhead, and poor user experience. By extending the interface to manage the complete data lifecycle and enabling in-place upgrades and rebuilds, Fluid can provide seamless, resilient, and efficient data management—improving system availability, reducing operational friction, and aligning with production-grade requirements for cloud-native data orchestration.

[Monthlyaway]

Hi @cheyang , here is a proposal for this feature request. Looking forward to join the LFX mentorship program!

Proposal: Extend Cache Runtime Interface for Full Data Lifecycle

1. Current Architecture Analysis

Fluid's current Dataset Lifecycle Management is primarily focused on the operations: provisioning and deletion. The architecture relies on a coordination between the Dataset Controller and specific Runtime Controllers (e.g., Alluxio, JuiceFS).

Key Components & Roles

• Dataset Controller (pkg/controllers/v1alpha1/dataset/dataset_controller.go):

  • Acts as the orchestrator. It watches Dataset CRs.
  • Creation: It validates the dataset and triggers the creation of the corresponding Runtime (if it doesn't exist) via deploy.ScaleoutRuntimeControllerOnDemand and utils.CreateRuntimeForReferenceDatasetIfNotExist.
  • Deletion: It handles the cleanup via finalizers (fluid-dataset-controller-finalizer), ensuring resources are released before the Dataset object is removed.
  • Status Sync: It updates the Dataset status (e.g., Bound, NotBound) but delegates deep runtime management to the Runtime Controller.

• Runtime Controller (e.g., pkg/controllers/v1alpha1/alluxio/alluxio_runtime_controller.go):

  • Reconciles the specific Runtime CR (e.g., AlluxioRuntime).
  • It uses the Engine Interface (pkg/ddc/base/engine.go) to interact with the underlying cache system.
  • Crucially, the shared logic resides in pkg/controllers/runtime_controller.go, specifically the ReconcileInternal and ReconcileRuntime methods.

• Template Engine (pkg/ddc/base/template_engine.go):

  • Most runtimes (Alluxio, Jindo, GooseFS) embed this struct.
  • It implements the Engine interface by delegating to an Implement interface.
  • Current Limitations: The interface supports Setup (provisioning), Sync (status updates/maintenance), and Shutdown (deletion). It lacks explicit methods operations like Upgrade (version change) or Rebuild (recovery from failure). Currently, changing an image version in the CRD often results in undefined behavior or requires a full delete-and-recreate cycle to take effect.

2. Proposed Architectural Changes

To support full data lifecycle management (specifically in-place upgrades and rebuilds), we need to evolve the Engine Interface from a stateless "Setup/Sync" model to a state-aware lifecycle model.

2.1 Extended State Machine

We propose introducing a new transitional phase to the Dataset status. Currently, the Dataset API supports Pending, Bound, NotBound, Failed, and Updating. We need to add:

• Recovering: The runtime is rebuilding lost components (e.g., after a node failure).

We will leverage the existing Updating phase for in-place upgrades.

+-----------+        Config/Image Change         +----------+
|           | ---------------------------------> |          |
|   Bound   |                                    | Updating |
|           | <--------------------------------- |          |
+-----------+        Upgrade Complete            +----------+
      ^
      | Failure Detected
      v
+-----------+        Rebuild Complete            +----------+
|           | <--------------------------------- |          |
|  Failed   |                                    | Recovering|
|           | ---------------------------------> |          |
+-----------+        Trigger Rebuild             +----------+

2.2 Interface Extension

We will extend the Engine and Implement interfaces in pkg/ddc/base/engine.go to support these new transitions.

• ShouldUpgrade() (bool, error): Checks if the current Runtime spec (e.g., image version) differs from the running state.
• Upgrade() error: Orchestrates the rolling upgrade of the runtime components (Master/Worker).
• ShouldRebuild() (bool, error): Checks if components are in an unrecoverable failed state that requires manual intervention.
• Rebuild() error: Triggers the reconstruction of specific components.

2.3 Controller Logic Flow

The shared RuntimeReconciler.ReconcileRuntime method (in pkg/controllers/runtime_controller.go) will be updated to include these checks before the standard Sync loop:

[ ReconcileRuntime ]
       |
       v
  [ Setup (if new) ]
       |
       v
[ Check ShouldUpgrade? ] --Yes--> [ Set Status: Updating ] --> [ Call Engine.Upgrade() ]
       |                                                                |
       No                                                               v
       |                                                        [ Upgrade Done? ] --Yes-> [ Set Status: Bound ]
       v
[ Check ShouldRebuild? ] --Yes--> [ Set Status: Recovering ] -> [ Call Engine.Rebuild() ]
       |
       No
       |
       v
  [ Sync (Standard) ]

3. Implementation Phases

Phase 1: Interface Definition & API Updates

• Goal: Define the contract for the new operations.
• Actions:
  • Modify api/v1alpha1/dataset_types.go to include the Recovering phase.
  • Update pkg/ddc/base/engine.go to add Upgrade, Rebuild, ShouldUpgrade, and ShouldRebuild methods.
  • Update pkg/ddc/base/template_engine.go to implement these methods (delegating to the embedded implementation).

Phase 2: Base Controller Integration

• Goal: Wire the new logic into the reconciliation loop.
• Actions:
  • Update pkg/controllers/runtime_controller.go: Modify the ReconcileRuntime function to invoke the new engine methods.
  • Implement the state transition logic: Update the Dataset status to Updating or Recovering before triggering the engine operations, and revert to Bound upon success.

Phase 3: Reference Implementation (ThinRuntime)

• Goal: Prove the design with a working example.
• Actions:
  • Implement the new methods in pkg/ddc/thin/implement.go.
  • Upgrade Logic: Implement a basic rolling update (e.g., updating the StatefulSet image field).
  • Rebuild Logic: Implement logic to delete and recreate failing pods.
  • Validate with a scenario: Deploy -> Change Image -> Verify Rolling Update.