[Q&A] Question about Snapshot Persistence in v2.7.0

[or9327]

Python version (python3 -V)

3.10.19

NVFlare version (python3 -m pip list | grep "nvflare")

2.7.0

NVFlare branch (if running examples, please use the branch that corresponds to the NVFlare version, git branch)

2.7

Operating system

Debian Linux (python:3.10-slim)

Have you successfully run any of the following examples?

• hello-numpy-sag with simulator
• hello-pt with simulator
• hello-numpy-sag with POC
• hello-pt with POC

Please describe your question

Context

We have successfully completed a PoC using NVFlare (https://github.com/NVIDIA/NVFlare/tree/main/examples/advanced/nlp-ner) using 2 clients and 1 server in Production mode and are planning to deploy it to production. Based on the FAQ documentation, I understand that:

• Load balancing between multiple FL servers is not currently supported
• If the FL server crashes, the running job will be interrupted and needs to be resubmitted

However, since federated learning jobs typically run across multiple rounds, I believe snapshot persistence could make NVFlare viable for production use by enabling job recovery from the last saved state.

Understanding of HA Mode Requirements

From my research, I understand that:

• Snapshot persistence requires ha_mode: true
• HA mode only works with HttpOverseerAgent (not DummyOverseerAgent)

The Problem

When attempting to configure HttpOverseerAgent in project.yml for v2.7.0, I've encountered what appears to be contradictory requirements:

Issue 1: Overseer cannot be defined as a participant

• In v2.7.0, it's no longer possible to define Overseer as a 'participant' in project.yml

Issue 2: StaticFileBuilder configuration doesn't propagate

• I added HttpOverseerAgent configuration to StaticFileBuilder in project.yml:

- path: nvflare.lighter.impl.static_file.StaticFileBuilder  
  args:  
    config_folder: config
    ha_mode: true  
    overseer_agent:  
      path: nvflare.ha.overseer_agent.HttpOverseerAgent  
      args:  
        overseer_end_point: http://100.64.0.40:8443/api/v1  
        project: ha-poc

• However, the generated fed_server.json still shows DummyOverseerAgent configuration
• If I manually modify fed_server.json to use HttpOverseerAgent, I get a "not secure content" error and the server startup kit becomes unusable

Issue 3: Unable to verify snapshot functionality

• Due to the above issues, I cannot verify whether snapshot persistence actually works in v2.7.0

Questions

1. Is snapshot persistence supported in NVFlare v2.7.0?
2. If, snapshot persistence is supported, is there official documentation or an example for setting up snapshot persistence?
3. If, snapshot persistence is supported, which environment is supported? - Server, Docker container, Kubernetes

Environment

NVFlare Version: 2.7.0
Python Version: 3.10.19
Deployment Platform: Google Kubernetes Engine (GKE)
OS: Debian-based (python:3.10-slim Docker image)

Key Dependencies:

• PyTorch: 2.1.2 (CPU)
• Transformers: 4.36.2
• Google Cloud Storage SDK

Deployment Configuration:

Server Startup Command for K8S:

python3 -u -m nvflare.private.fed.app.server.server_train \
  -m /workspace/nvfl/server1-svc-ex.nvflare.svc.cluster.local \
  -s fed_server.json \
  --set secure_train=true config_folder=config org=kai ha_mode=true

Current fed_server.json (Auto-generated):

{
  "format_version": 2,
  "servers": [...],
  "overseer_agent": {
    "path": "nvflare.ha.dummy_overseer_agent.DummyOverseerAgent",
    "args": {
      "sp_end_point": "server1-svc-ha.nvflare.svc.cluster.local:8002:8003"
    }
  }
}

Snapshot Persistor Configuration (in resources.json.default):

{
  "snapshot_persistor": {
    "path": "nvflare.app_common.state_persistors.storage_state_persistor.StorageStatePersistor",
    "args": {
      "uri_root": "/",
      "storage": {
        "path": "nvflare.app_common.storages.filesystem_storage.FilesystemStorage",
        "args": {
          "root_dir": "/workspace/nvflare/snapshot-storage",
          "uri_root": "/"
        }
      }
    }
  }
}

---

Any guidance on the proper configuration approach would be greatly appreciated!

---

[YuanTingHsieh]

@or9327 thank you so much for the information!

We deprecated our current support for HA as that is not complete.

For your use case, on the controller side you can add saving mechanism (for example "torch.save") to save the aggregated model parameters after each round.
So if the job unfortunately failed, those model weights are preserved.

remember to also read these saved model back to start the next round.