IAM authentication support for Redis, Kafka, S3 and PG

[n1ru4l]

Enterprise self-hosters have the requirement of using IAM authentication for the following services:

• Redis
• Kafka
• S3
• Postgres

The configuration for these options should be done via the existing environment variables prefixes, e.g. POSTGRES_*, REDIS_*, KAFKA_* and S3_*.

---

Resources:

• ElastiCache (Redis)
• S3
• MSK (Kafka)
• RDS (Postgres)

[mish-elle]

Hi team! We’re looking into this now, and here are our current thoughts on the implementation approach.

1. S3 - We plan to use the AWS SDK credential provider chain for S3 credentials. The aws4fetch client already supports session tokens.
2. Postgres - We plan to generate RDS IAM authentication tokens and use them as the database password. We will also implement token refresh so long-running deployments do not require restarts.
3. Redis - We plan to follow the same pattern as Postgres, including generating credentials/tokens as required and handling refresh.
4. MSK - We plan to use KafkaJS's oauthbearer SASL mechanism with aws-msk-iam-sasl-signer-js to generate short-lived SigV4-signed tokens. KafkaJS re-invokes the token provider on each broker connection, so refresh should be automatic.

We can implement the shared AWS/IAM auth mechanisms as a shared package, per your recommendation. Would you prefer we extend the existing service-commons package for this, or would you rather we introduce a new shared package (for example internal/aws, similar to internal/postgres) and have the relevant services depend on it?

Other than S3, the following env vars will be introduced

1. If AWS_REGION is set (set by AWS), then IAM Auth is enabled.
2. Each service will have _AWS_REGION, to allow users to override the aws region for a service.
3. Each service will have _IAM_AUTH_ENABLED, which is true by default, so that users can turn off IAM AUTH for specific services when deployed on AWS.

Since S3 uses the credential provider chain, we'll give the highest priority to static credentials (S3_ACCESS_KEY_ID and S3_SECRET_ACCESS_KEY) if these are not provided then the credential provider chain will use IAM AUTH. And S3's region is specified via the S3_ENDPOINT.

[n1ru4l]

@mish-elle great to have you on board!

I have no concerns regarding the shared packages.

Regarding the location of the code. I think the service-common package is a good starting point. We could later on break it down into it's own or even more isolated packages if needed, though that should not be a concern for you at the moment.

To summarise the flows:

pg/redis/kafka (long-lived connections)

flowchart TD
    A[Service] --> B["Generate IAM Auth Token
(SigV4, ~15 min TTL)"]
    B --> C["Connect to RDS / Redis
Username + Token as Password (TLS)"]
    C --> D["Service Validates Token
IAM Policy + Signature"]
    D --> E[Connection Established]

    E --> F[Connection Ends / Drops]
    F --> G[Token Expired or Not Reusable]
    G --> H[Generate NEW IAM Token Required]

    H --> B

Loading

I assume that each service replica requests and manages its own IAM credentials generation?

S3 (single HTTP requests)

flowchart TD
    A[Service] --> B[Prepare S3 Request GET PUT etc]
    B --> C[Sign request with IAM credentials SigV4]
    C --> D[Send HTTPS request to S3]
    D --> E[S3 validates signature]
    E --> F[Evaluate IAM policy and bucket policy]

    F -->|Allowed| G[Request succeeds]
    F -->|Denied| H[Access denied]

    G --> I[Next request requires new signature]
    I --> B

Loading

Service environment variables

To confirm your proposed plan the following environment variables will be introduced?

# global environment variable that propagates to each db type 
AWS_REGION
# override for global AWS_REGION on each db
(POSTGRES|REDIS|KAFKA)_AWS_REGION
# opt-out a specific db from using IAM
(S3|POSTGRES|REDIS|KAFKA)_IAM_AUTH_ENABLED

There will be no S3_AWS_REGION environment variable, as the region for that is instead extracted from the S3_ENDPOINT environment variable (https://s3.{region}.amazonaws.com)?

[mish-elle]

"I assume that each service replica requests and manages its own IAM credentials generation?"
Yes! These flow diagrams are aligned with our current implementation approach.

Yes, on the new env vars and no S3_AWS_REGION.
We can also update the IAM_AUTH env var name to specify it is for AWS,

# global environment variable that propagates to each db type 
AWS_REGION
# override for global AWS_REGION on each db
(POSTGRES|REDIS|KAFKA)_AWS_REGION
# opt-out a specific db from using IAM
(S3|POSTGRES|REDIS|KAFKA)_AWS_IAM_AUTH_ENABLED

Since there are a few S3 clients that get configured we can also support

S3_AWS_IAM_AUTH_ENABLED
S3_MIRROR_AWS_IAM_AUTH_ENABLED
S3_AUDIT_LOG_AWS_IAM_AUTH_ENABLED

Credential Resolutions

This would be a breaking change if someone is deployed on AWS and does not have IAM setup.

S3

flowchart TD
    A[Resolve S3 Credentials] --> B{AWS_REGION set?}
    B -->|Yes| C{S3_AWS_IAM_AUTH_ENABLED = false?}
    C -->|No / not set| D[✅ SDK credential chain<br/>IAM auth]
    C -->|Yes| E{Static creds provided?<br/>S3_ACCESS_KEY_ID +<br/>S3_SECRET_ACCESS_KEY}
    B -->|No| E
    E -->|Yes| F[✅ Use static credentials]
    E -->|No| G[🚨 Throw: static creds required]

Loading

MSK

flowchart TD
    A[Resolve Kafka SASL] --> B{AWS_REGION set?}
    B -->|Yes| C{KAFKA_AWS_IAM_AUTH_ENABLED = false?}
    C -->|No / not set| D[✅ OAUTHBEARER + SigV4 token<br/>Region: KAFKA_AWS_REGION ?? AWS_REGION]
    C -->|Yes| E{Explicit SASL mechanism?<br/>plain / scram-sha-*}
    B -->|No| E
    E -->|Yes| F[✅ Use username + password]
    E -->|No| G[✅ No SASL — unauthenticated]

Loading

RDS

flowchart TD
    A[Resolve Postgres Auth] --> B{AWS_REGION set?}
    B -->|Yes| C{POSTGRES_AWS_IAM_AUTH_ENABLED = false?}
    C -->|No / not set| D[✅ Generate RDS IAM auth token<br/>+ refresh on expiry<br/>Region: POSTGRES_AWS_REGION ?? AWS_REGION]
    C -->|Yes| E{Static password provided?<br/>POSTGRES_PASSWORD}
    B -->|No| E
    E -->|Yes| F[✅ Use static password]
    E -->|No| G[🚨 Throw: password required]

Loading

Redis

flowchart TD
    A[Resolve Redis auth config] --> B{AWS_REGION set?}
    B -->|Yes| C{REDIS_AWS_IAM_AUTH_ENABLED = false?}
    C -->|No / not set| D[✅ Generate ElastiCache IAM auth token<br/>+ refresh on expiry<br/>Region: REDIS_AWS_REGION ?? AWS_REGION]
    C -->|Yes| E{REDIS_PASSWORD provided?}
    B -->|No| E
    E -->|Yes| F[✅ Connect with static password]
    E -->|No| G[✅ Connect with empty password]

Loading

[n1ru4l]

Thanks these diagrams are very useful! In order to avoid the breaking change for S3, the *_AWS_IAM_AUTH_ENABLED environment variables could default to false instead? 🤔 It would only be breaking if an AWS_REGION environment variable already existed? I prefer to be more explicit instead of deciding based on AWS_REGION whether iam auth is enabled or not.

I agree that these should also be introduced for S3 mirror/audit logs environment variables, as the mirror could be AWS or another provider, such as R2 for going multi-cloud.

[mish-elle]

"It would only be breaking if an AWS_REGION environment variable already existed?"
Yes, but we can set *_AWS_IAM_AUTH_ENABLED to false for all services by default so there's no breaking changes 😄

[mish-elle]

Hey @n1ru4l while working on Redis and RDS changes, we're looking for some feedback on the token refresh strategy

Elasticache

For context, ElastiCache IAM auth uses SigV4 pre-signed tokens as passwords, which expire every 15 minutes and need periodic rotation. The challenge is that ioredis doesn't accept a function/provider as its password option, it only takes a static string. So we need a strategy for rotating credentials on live connections.

Approach A: In-place re-authentication

Our current implementation:

1. Generates a SigV4 token and passes it as the initial password to the ioredis constructor
2. Sets up a timer that fires every 12 minutes (with jitter) to generate a fresh token
3. Sends the Redis AUTH command on the live connection to re-authenticate
4. Mutates redis.options.password (and connectionPool.redisOptions.password for cluster mode) so any future reconnects use the new token

export async function refreshIamAuth(
  redis: RedisInstance,
  token: string,
  username: string,
  isCluster: boolean,
  label: string,
  logger: AwsIamLogger,
): Promise<void> {
  if (isCluster) {
    const nodes = (redis as any).nodes?.('all') ?? [];
    logger.info(
      '%s: Refreshing IAM token (service=elasticache) — re-authenticating %s cluster node(s)',
      label,
      nodes.length,
    );
    for (const node of nodes) {
      try {
        await node.call('AUTH', username, token);
        node.options.password = token;
      } catch (err) {
        logger.warn('%s: Failed to re-AUTH cluster node (service=elasticache, error=%s)', label, err);
      }
    }
    // Update cluster-level redisOptions for new node connections.
    const pool = (redis as any).connectionPool;
    if (pool?.redisOptions) {
      pool.redisOptions.password = token;
    }
  } else {
    await (redis as any).call('AUTH', username, token);
    (redis as any).options.password = token;
  }
  logger.info('%s: IAM token refreshed successfully (service=elasticache)', label);
}

Approach B: Client swap via proxy (with overlap)

Since we come from a Java/JVM background, in Java credential fields on connection objects are private and immutable, you can't mutate them post-construction. So the established approach is:

1. Proxy wrapper: Create a thin wrapper class that implements the same Redis interface and delegates all commands to an inner client reference
2. Early rotation: ~5 minutes before the token expires, construct a new client with fresh credentials while the old client continues serving inflight requests
3. Swap: Once the new client is connected and ready, atomically swap the inner reference in the wrapper so all subsequent commands route to the new client
4. Teardown: Gracefully close the old client after inflight commands drain

This gives you credential rotation using only public APIs, with no reliance on internal framework state. The proxy is transparent to consumers, they hold a reference to the wrapper and will never know a swap happened.

Approach C: ioredis to node redis

I believe node redis has support for function passwords, however I think that swapping redis clients would be out of scope for this issue.

RDS

Similar to Redis, the challenge is the same as slonik's createPool only accepts a static connection string. Similar approaches in which we either (A) mutate the underlying connectionString or (B) create a new pool with the new connectionString.

Notes: MSK and S3

KafkaJS natively supports credential rotation via the oauthBearerProvider callback in its SASL configuration. The provider function is invoked every time KafkaJS needs to authenticate a broker connection (including reconnects).

S3 access signs each HTTP request individually with SigV4 at request time. So there is no need to handle token refreshes.

What we'd like feedback on

1. Redis: Are you comfortable with the in-place AUTH approach (A) given the reliance on ioredis internals, or would you prefer the client swap pattern (B)?
2. PostgreSQL: Do you prefer the internal mutation (A) or the pool swap (B)?

[mish-elle]

Some additional updates on Redis IAM Auth

Additional Environment Variables

In addition to REDIS_AWS_IAM_AUTH_ENABLED and REDIS_AWS_REGION, we also need to introduce

Variable	Required	Default	Description
REDIS_USERNAME	No	'default'	ElastiCache username for IAM auth. ElastiCache uses 'default' as the built-in user
REDIS_AWS_IAM_CACHE_NAME	No	—	ElastiCache cache name. Used as the hostname in the SigV4 presigned token (required by ElastiCache IAM to identify the target cache)

---
title: Redis (Long-lived Connection)
---
flowchart TD
    A[createRedisClient] --> B{"AWS_REGION set?"}
    B -->|Yes| C{"REDIS_AWS_IAM_AUTH_ENABLED = '1'?"}
    C -->|Yes| VALIDATE
    B -->|No| PW["✅ Use static REDIS_PASSWORD"]
    C -->|No| PW

    subgraph VALIDATE ["Validate Required IAM Vars"]
        direction TB
        V1["Check REDIS_TLS_ENABLED"]
        V2["Check REDIS_AWS_IAM_CACHE_NAME"]
        V1 & V2 -->|any missing| X["❌ Throw: Redis IAM authentication requires:<br/>REDIS_TLS_ENABLED, REDIS_AWS_IAM_CACHE_NAME"]
    end

    VALIDATE -->|all set| DEFAULTS

    subgraph DEFAULTS ["Resolve Defaults"]
        D1["REDIS_USERNAME<br/>⚠️ optional — defaults to 'default'"]
        D2["awsRegion = REDIS_AWS_REGION ?? AWS_REGION"]
    end

    style X fill:#ff6b6b,color:#fff
    style D1 fill:#ffd43b,color:#000
    style D2 fill:#ffd43b,color:#000

Loading

Redis Cluster Mode Support

The current codebase only supports standalone Redis. In AWS, ElastiCache cluster mode is the typical production deployment for higher availability and throughput. So we're looking to add support for cluster mode for our use case.

Would you like us to include cluster mode support in this issue, or should it be tracked separately?

[n1ru4l]

Regarding Redis IAM, there is a pending PR to support providing the password as a function:

• feat(auth): add support for function for password redis/ioredis#1985

And also a (dirty) workaround

• Is credential provider supported? redis/ioredis#1738

Approach A seems fine with me, until proper support has landed. It also seems like there is an opportunity to take over the pending PR and fix the tests so it can be merged.

Regarding RDS:

• Support for this could be natively added within slonik
• As a quick solution it could also be added via pnpm patch package (we already use it in slonik anyways for disabling noisy tracing)
• Creating a new pool, seems not like the optimal solution, since individual clients do the actual authentication not the pool as a whole

So I recommend patching slonik for now, then implement support for dynamic passwords within slonik.

[n1ru4l]

Regarding the Redis Cluster Mode:

For us we did not see the need to use replicated Redis yet, it never became a bottle neck, though it is definitely something we will consider, especially since it is used for caching tokens for usage reporting.

For me it is okay to track this as part of this issue or a new issue, depending on what works best for you. In any case it makes sense to think about the configuration for IAM with redis cluster mode in mind.

[mish-elle]

FYI for ioredis, there's a follow up PR that fixes the tests,

• feat(auth): add support for function for password redis/ioredis#2039

but it looks like it is being held off for now as they are evaluating adding an auth flow similar in node-redis.

Yes, that sounds good for RDS, thanks!

[mish-elle]

Hey @n1ru4l while working on cluster support for redis, we noticed that here's a bug with bentocache's L2 cache support for cluster that was fixed in this issue.

• Redis cluster not supported Julien-R44/bentocache#87

The fix shipped in bentocache v1.5.1. We've been working around it with a pnpm patch, but upgrading to v1.5.1 would resolve it properly. Would it be possible to bump the bentocache dependency? I see it's tracked on the Dependencies Dashboard.

[n1ru4l]

@mish-elle Here you go #8019