Skip to content

DEPLOYMENT.md

Latest commit

 

History

History
571 lines (428 loc) · 19 KB

DEPLOYMENT.md

File metadata and controls

571 lines (428 loc) · 19 KB

Deployment Guide

This document explains the deployment architecture and configuration for the Codeware monorepo, which uses GitHub Actions to automatically deploy applications to Fly.io with support for multi-tenant deployments.

Table of Contents

Overview

The deployment system automatically:

  • Determines deployment environment (preview for PRs, production for main)
  • Analyzes which Nx apps have been affected by code changes
  • Fetches tenant configuration from Infisical for multi-tenant apps
  • Deploys each app to Fly.io (once per tenant for multi-tenant apps)
  • Posts preview URLs as PR comments

Manual Redeployment

Sometimes deployments fail or you need to redeploy without pushing new code. The workflow supports manual triggers via GitHub Actions UI.

How to Trigger Manual Deployment

  1. Navigate to Actions → Fly Deployment in GitHub
  2. Click Run workflow button
  3. Configure deployment options:
    • App: Select specific app (cms, web) or leave empty for all affected apps
    • Tenant: Enter tenant ID (e.g., demo) or leave empty for all tenants
    • Environment: Choose preview or production (required)
    • Console logs: Enable to see aggregated application logs during deployment (useful for debugging Fly commands)
  4. Click Run workflow

Use Cases

Redeploy everything to production:

  • App: <empty>
  • Tenant: <empty>
  • Environment: production

Redeploy specific app for all tenants:

  • App: web
  • Tenant: <empty>
  • Environment: production

Redeploy specific tenant:

  • App: web
  • Tenant: acme
  • Environment: production

Redeploy CMS only:

  • App: cms
  • Tenant: <empty>
  • Environment: production

Debug deployment CLI issues:

  • App: web
  • Tenant: demo
  • Environment: preview
  • Console logs: enabled

Note

Manual deployments bypass the affected app analysis and deploy the specified app(s) regardless of code changes. The tenant input only applies to multi-tenant apps like web.

Architecture

Components 

┌──────────────────────────────────────┐
│ GitHub Actions Workflow              │
│                                      │
│ .github/workflows/fly-deployment.yml │
└──────────────────────────────────────┘
    │
    ├─► Job 1: analyze-conditions
    │    ├─ Analyze required deploy conditions
    │    │   ├─ Skip Renovate workflows
    │    │   ├─ Skip Nx Migrate workflows
    │    │   └─ Skip Pull Request updates unless the preview label exists
    │    ├─ Detect a Pull Request is opened or reopened
    │    │   └─ Add label 'preview-deploy'
    │    └─ Output: skip
    │
    ├─► Job 2: pre-deploy
    │    ├─ Abort deployment process when skip output from job 1 is 'true'
    │    ├─ Determine environment (preview/production)
    │    ├─ Analyze affected Nx apps
    │    ├─ Fetch secrets and app-tenant relationships
    │    │   from Infisical
    │    └─ Output: apps, environment, app-tenants
    │
    └─► Job 3: fly-deployment
         ├─ For each app to deploy:
         │   ├─ If multi-tenant: deploy once per tenant
         │   └─ If single-tenant: deploy once
         └─ Post preview comment (for PRs)

Key Packages

  1. @cdwr/nx-pre-deploy-action - Analyzes deployment requirements
    • Determines target environment based on GitHub event
    • Identifies affected Nx applications
    • Validates github.json for each application
    • Fetches app-specific tenant configuration and secrets from Infisical
  2. @cdwr/nx-fly-deployment-action - Executes deployments
    • Manages Fly.io application lifecycle
    • Handles multi-tenant deployments
    • Manages preview/production environments

Note

See individual package READMEs for detailed API documentation, configuration options, and usage examples.

Configuration

Per-App Configuration (github.json)

Each deployable app needs a github.json file in its root directory (same location as fly.toml):

{
  "$schema": "../../libs/shared/util/schemas/src/lib/github-config.schema.json",
  "flyPostgresPreview": "${POSTGRES_PREVIEW}",
  "flyPostgresProduction": "my-production-db",
  "flyPostgresDatabaseName": "shared_database"
}

Fields:

  • flyPostgresPreview (string, optional) - Fly Postgres cluster name for preview env
  • flyPostgresProduction (string, optional) - Fly Postgres cluster name for production env
  • flyPostgresDatabaseName (string, optional) - Shared database name for all apps (ensures multiple apps use the same database instead of creating separate ones)

Deployment Detection:

Apps are automatically detected for deployment if they have:

  1. A github.json file in the app root
  2. A Fly configuration file (see Fly Configuration Files)

Examples:

Multi-tenant app (web):

{}

Empty github.json is valid - the app will be deployed if it has a fly.toml file.

Single-tenant app (cms):

Using a Fly Postgres cluster for preview (pull request) apps. Production database is not hosted by Fly.

{
  "flyPostgresPreview": "${POSTGRES_PREVIEW}",
  "flyPostgresDatabaseName": "cdwr_cms_shared"
}

Note: flyPostgresDatabaseName ensures that both the platform CMS host and all tenant deployments share the same database. Without this, each app would get its own empty database, causing "relation does not exist" errors in tenant apps.

Fly Configuration Files

Apps need a Fly configuration file to be deployed. The system looks for config files in this priority order:

  1. Environment-specific config: fly.{environment}.toml (e.g., fly.production.toml, fly.preview.toml)
  2. Default config: fly.toml

This allows you to have different Fly configurations per environment while falling back to a shared config when environment-specific files don't exist.

Important

Fly configuration files are only used for deployment of new apps. For existing apps the remote configurations are preserved.

This is to prevent overriding any individual ad-hoc configurations applied to the apps.

Example: Different machine sizes per environment

# fly.preview.toml - Smaller machines for preview
app = "my-app"

[[vm]]
  size = 'shared-cpu-1x'
  memory = '512mb'
# fly.production.toml - Larger machines for production
app = "my-app"

[[vm]]
  size = 'shared-cpu-2x'
  memory = '2gb'

Tenant Configuration (Infisical)

Infisical is the single source of truth for both secrets and tenant configuration.

Key Concepts:

  • App-level secrets: /apps/<app-name>/*
  • Tenant-app secrets: /tenants/<tenant-id>/apps/<app-name>/*
  • Tenant discovery: System scans /tenants/ folder structure to determine which tenants use which apps
  • Dynamic CORS: CMS automatically fetches tenant app URLs tagged with cors at boot for CORS configuration

[!NOTE] Detailed multi-tenant setup

  • Complete folder structure guide
  • Secret classification (env vars vs encrypted secrets)
  • Step-by-step configuration examples
  • Deploy rules configuration

See: Multi-tenant Setup Guide

Quick Example:

# App-wide configuration
/apps/web/API_URL = "https://api.example.com"

# Per-tenant configuration
/tenants/demo/apps/web/PUBLIC_URL = "https://demo.example.com"
/tenants/acme/apps/web/PUBLIC_URL = "https://acme.example.com"

# Hybrid deployment (both cms host + tenant-scoped)
/tenants/_default/apps/cms/  # CMS host (no TENANT_ID)
/tenants/demo/apps/cms/      # Tenant-scoped CMS (TENANT_ID=demo)

Tip

Use the reserved tenant name _default to be able to deploy an app both as a cms host instance (without TENANT_ID) and as tenant-scoped instances. The _default tenant follows DEPLOY_RULES like any other tenant.

Secret Loading: Deployment vs Runtime

Secrets are loaded at two distinct stages, each serving different purposes:

Deployment-Time Secrets (loaded during GitHub Actions)

  • Location: /tenants/<tenant-id>/apps/<app-name>/ (recursive)
  • Purpose: Configuration that determines how apps are deployed
  • Fetched by: Pre-deploy action during CI/CD workflow
  • Examples: CUSTOM_URL, PAYLOAD_API_KEY, tenant-specific build configuration
  • Characteristics:
    • Baked into Fly.io app configuration as env vars or secrets
    • Static after deployment (requires redeployment to change)
    • Uses metadata env: true to distinguish env vars from secrets (default)
  • Use when: Configuration defines tenant-specific deployment details

Runtime Secrets (loaded when app starts)

  • Location:
    • /apps/<app-name>/ (recursive)
    • /tenants/<tenant-id>/ (shallow - to avoid loading tenants apps secrets)
  • Purpose: Sensitive data and operational secrets
  • Fetched by: Application itself at startup using withInfisical() SDK
  • Examples: Database credentials, API keys, encryption keys, feature flags
  • Characteristics:
    • Loaded fresh on each app startup
    • Can be rotated without redeployment (app restart required)
    • Requires Infisical credentials set as Fly.io secrets
  • Use when: Secrets need frequent rotation or shouldn't be bundled in the deployment

Key Limitation: Deployment-time secrets are static until next deployment. Runtime secrets add startup latency but enable rotation without redeployment.

Public or hidden secret:

Secrets can be resolved to either an environment variable or a hidden secret in Fly. Environment variables are visible and added to the Docker image at build-time and secrets are loaded at boot-time, fully encrypted.

Secrets in Infisical are handled as secrets by default.

To make a secret visible as environment variable, add metadata key env set to true.

Sentry Releases

Sentry releases enable you to associate errors with specific deployments, track which versions have bugs, and get better source map resolution.

How it works:

  1. Release creation (GitHub workflow): A single release is created using the Git SHA before any deployments
  2. Commit association: The release is linked to commits for better error tracking
  3. Source map upload (Docker build): Each app uploads source maps to the shared release during build
  4. Release finalization (GitHub workflow): After all deployments succeed, the release is finalized and marked as deployed
  5. Error tracking: Errors are automatically associated with the release version

Workflow Steps:

The deployment workflow (.github/workflows/fly-deployment.yml) orchestrates the release lifecycle:

Docker Build Requirements:

For source maps to upload during Docker builds, the Sentry environment variables must be:

  1. Passed as build arguments (--build-arg) to the docker build command
  2. Converted to environment variables (ENV) in the Dockerfile before the build command

Example in Dockerfile:

ARG SENTRY_AUTH_TOKEN
ARG SENTRY_ORG
ARG SENTRY_PROJECT
ARG SENTRY_RELEASE

# Convert to ENV so Sentry webpack plugin can access them during build
ENV SENTRY_AUTH_TOKEN=$SENTRY_AUTH_TOKEN \
    SENTRY_ORG=$SENTRY_ORG \
    SENTRY_PROJECT=$SENTRY_PROJECT \
    SENTRY_RELEASE=$SENTRY_RELEASE

RUN npx nx build cms  # Source maps uploaded here

Important

With multi-stage builds, these ENV variables exist only in the builder stage and are not included in the final deployed image.

Deployment Rules (Required)

Control which apps and tenants are deployed per environment using a DEPLOY_RULES secret in the Infisical root path (/). This secret is required - deployments will fail if it's missing or misconfigured.

Quick Setup:

Create a DEPLOY_RULES secret with metadata or JSON value per environment:

Path: /
Secret: DEPLOY_RULES
Metadata per environment:
  preview:
    apps: '*'
    tenants: 'demo' # Only demo tenant in preview
  production:
    apps: '*'
    tenants: '*' # All tenants in production

Rules format: apps: "*" (all apps) or apps: "web,cms" (specific apps) | tenants: "*" (all) or tenants: "demo,acme" (specific)

[!NOTE] Complete deployment rules documentation

  • Both configuration options (metadata vs JSON value)
  • Full rules format reference
  • Validation requirements
  • Advanced use cases

See: Deploy Rules Documentation

GitHub Secrets

Required secrets in GitHub repository settings:

INFISICAL_READ_CLIENT_ID       # Infisical auth client ID
INFISICAL_READ_CLIENT_SECRET   # Infisical auth client secret
INFISICAL_PROJECT_ID           # Infisical project ID
FLY_API_TOKEN                  # Fly.io API token
CDWR_ACTIONS_BOT_ID            # GitHub App ID for bot
CDWR_ACTIONS_BOT_PRIVATE_KEY   # GitHub App private key

Required variables:

INFISICAL_SITE                 # Infisical site region
FLY_ORG                        # Fly.io organization
FLY_REGION                     # Fly.io default region
FLY_OPT_OUT_DEPOT              # Opt out of depot builder
FLY_POSTGRES_PREVIEW           # Preview postgres cluster

Deployment Flow

sequenceDiagram
    participant Evt as Event
    participant GH as GitHub Workflow
    participant Pre as Pre-deploy Action
    participant Dep as Fly-deploy Action
    participant Inf as Infisical
    participant Fly as Fly.io

    Evt->>GH: PR opened/updated
    Evt->>GH: PR closed
    Evt->>GH: Push to main
    GH->>Pre: Analyze deployment
    alt Pull Request
      Pre->>Pre: Environment = preview
    else Push to main
      Pre->>Pre:Environment = production
    end

    Pre->>Pre: Analyze affected apps (Nx)
    Pre->>Pre: Verify github.json for apps
    Pre->>Inf: Fetch tenant details for apps to deploy
    Inf-->>Pre: Return app-tenant mapping
    Pre->>Pre: Analyze app-tenant relations
    Pre-->>GH: environment, app-tenants

    GH->>Dep: environment, app-tenants
    Dep->>Dep: Deploy or Destroy apps

    loop Deploy: For each app and tenant
      Dep->>Dep: Extract app-tenant relations
      Dep->>Dep: Lookup tenant secrets for app
      Dep->>Dep: Get app base name from fly.yml
      alt Preview: Multi-tenant app
        Dep->>Dep: Name: app-pr123-tenant1
      else Preview: Single-tenant app
        Dep->>Dep: Name: app-pr123
      else Production: Multi-tenant app
        Dep->>Dep: Name: app-tenant1
      else Production: Single-tenant app
        Dep->>Dep: Name: app
      end
          Dep->>Fly: Deploy app with env
          Fly-->>Dep: Verify deployment
          Dep->>Fly: Attach secrets to app
    end

    loop Destroy: For each app
      Dep->>Fly: Detach from Postgres cluster
      Fly-->>Dep: Verify detached app
      Dep->>Fly: Destroy app
    end

    alt Preview
      Dep-->>Evt: Post preview URLs as comment to PR
    end
Loading

Multi-Tenant Deployment

How It Works

  1. Tenant-App Relations: Pre-deploy action fetches tenant configuration from Infisical
  2. Per-Tenant Deployment: Each tenant gets its own isolated Fly.io app instance
  3. Naming Convention: The base app name from fly.toml (app = "base-name") is used with suffixes:
    • Production: <base-name>-<tenant-id>
    • Preview: <base-name>-pr<number>-<tenant-id>
  4. Environment Variables: Each instance receives TENANT_ID, DEPLOY_ENV, APP_NAME, and PR_NUMBER

Note

  • Detailed deployment mechanics
  • Naming conventions
  • app-details structure

See: Nx Fly Deployment Action - app-details input

Benefits

✅ Complete Isolation: Each tenant has its own app instance ✅ Independent Scaling: Scale tenants independently ✅ Tenant-Specific Configuration: Each tenant can have its own secrets/config ✅ Easy Rollback: Roll back one tenant without affecting others ✅ Clear Monitoring: Per-tenant metrics and logs

Single-Tenant Apps

  • Deploy once per environment
  • No tenant suffix in app name
  • Traditional deployment model
  • Example: cms app serves all tenants from single instance

How to Add a New App

  1. Create github.json in the app root:

    {}

    Or with Postgres configuration:

    {
  "flyPostgresPreview": "${POSTGRES_PREVIEW}",
  "flyPostgresProduction": "my-production-db"
}

  2. Create fly.toml in the app root:

    app = "my-new-app"
primary_region = "arn"

[build]
  dockerfile = "Dockerfile"

[http_service]
  internal_port = 3000
  force_https = true
  auto_stop_machines = 'suspend'
  auto_start_machines = true
  min_machines_running = 0
  processes = ['app']

[[vm]]
  size = 'shared-cpu-1x'
  memory = '1gb'

    Optionally create environment-specific configs:

    • fly.preview.toml - For preview deployments
    • fly.production.toml - For production deployments

  3. Configure Infisical:

    • Create a folder and secrets at /apps/my-new-app/*
    • For multi-tenant apps: Create folders for related tenants /tenants/<id>/apps/my-new-app/

  4. Push changes - deployment happens automatically!

How to Add/Remove Tenants

Add a New Tenant

  1. Create folder structure in Infisical for each app the tenant will use:

    /tenants/<new-tenant-id>/apps/<app>

  2. Add tenant-specific secrets when needed:

    /tenants/<new-tenant-id>/apps/web/API_KEY = "..."
/tenants/<new-tenant-id>/apps/web/CUSTOM_CONFIG = "..."

  3. Deploy - next deployment will automatically detect and deploy the new tenant!

Remove a Tenant

  1. Delete folder in Infisical:

    • Remove /tenants/<tenant-id>/ folder entirely

  2. Clean up Fly.io apps (manual):

    fly apps destroy web-<tenant-id> --yes
fly apps destroy web-pr123-<tenant-id> --yes  # if preview exists

  3. Next deployment will no longer include this tenant