Add ElastiCache auto-discovery for Valkey/Redis services by angelvilardellperez · Pull Request #5303 · percona/pmm

angelvilardellperez · 2026-04-29T13:40:06Z

Summary

Adds ElastiCache auto-discovery to PMM, following the same pattern as the existing RDS discovery feature. ElastiCache replication groups tagged with pmm_enable=true are automatically discovered and registered as Valkey services in PMM inventory.

Features

Background auto-discovery: Reconciler runs every 5 minutes, scanning AWS regions for tagged ElastiCache clusters
API endpoint: POST /v1/management/services:discoverElastiCache for manual discovery via Swagger/UI
Cluster Mode support: Uses ConfigurationEndpoint for Cluster Mode Enabled, PrimaryEndpoint + ReaderEndpoint for Cluster Mode Disabled
Auth filtering: Automatically skips clusters with AUTH tokens or ACL user groups (no credentials support yet)
Environment tag: Reads the AWS Environment tag to populate the PMM environment label
Stale removal: Services are removed from PMM when the pmm_enable=true tag is removed from the cluster
New node type: RemoteElastiCacheNode for inventory tracking (same pattern as RemoteRDSNode)

New files

File	Description
`api/management/v1/elasticache.proto`	Discovery API messages (DiscoverElastiCache, AddElastiCacheService)
`managed/services/management/elasticache.go`	API discovery + add service handler
`managed/services/management/elasticache_discovery.go`	Background auto-discovery reconciler

Modified files

api/management/v1/service.proto — Added DiscoverElastiCache RPC, elasticache to AddService/Response
api/inventory/v1/nodes.proto — Added RemoteElastiCacheNode type
managed/models/node_model.go — Added RemoteElastiCacheNodeType
managed/services/converters.go — Node type conversion for ElastiCache
managed/services/inventory/ — Node list/get/add support
managed/services/management/service.go — AddService routing + RemoveService cleanup
managed/cmd/pmm-managed/main.go — Background job startup

How it works

PMM starts → 30s delay → reconciler starts (every 5 min):
  1. DescribeReplicationGroups across all AWS regions (parallel)
  2. Filter: engine=redis|valkey, status=available, no AUTH/ACL
  3. ListTagsForResource → keep only pmm_enable=true
  4. Diff with existing PMM services (labeled managed_by=elasticache-autodiscovery)
  5. Add new → RemoteElastiCacheNode + ValkeyService + ValkeyExporter
  6. Remove stale → service + agents + node cleanup

AWS permissions required

elasticache:DescribeReplicationGroups
elasticache:ListTagsForResource

Uses the default AWS credential chain (IRSA on EKS, env vars, ~/.aws).

Testing

Tested locally with PMM dev container against AWS staging ElastiCache clusters
Verified: discovery, tag filtering, auth filtering, cluster mode enabled/disabled, add, remove, environment tag extraction

Labels applied to auto-discovered services

managed_by: elasticache-autodiscovery
source: elasticache
engine: valkey|redis
role: primary|reader|cluster

barryib · 2026-04-29T16:09:31Z

Resolves #5300

codecov · 2026-04-30T20:17:25Z

Codecov Report

❌ Patch coverage is 18.63118% with 428 lines in your changes missing coverage. Please review.
✅ Project coverage is 42.90%. Comparing base (85387c6) to head (3aba34e).

Files with missing lines	Patch %	Lines
...naged/services/management/elasticache_discovery.go	24.22%	209 Missing and 10 partials ⚠️
managed/services/management/elasticache.go	1.07%	184 Missing ⚠️
managed/services/converters.go	0.00%	11 Missing ⚠️
managed/cmd/pmm-managed/main.go	0.00%	5 Missing ⚠️
managed/services/inventory/grpc/nodes_server.go	0.00%	4 Missing ⚠️
managed/services/inventory/nodes.go	92.30%	1 Missing and 1 partial ⚠️
managed/services/management/service.go	33.33%	2 Missing ⚠️
managed/models/agent_helpers.go	0.00%	1 Missing ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##               v3    #5303      +/-   ##
==========================================
- Coverage   43.21%   42.90%   -0.31%     
==========================================
  Files         413      415       +2     
  Lines       42302    42826     +524     
==========================================
+ Hits        18280    18376      +96     
- Misses      22155    22572     +417     
- Partials     1867     1878      +11

Flag	Coverage Δ
admin	`34.94% <ø> (ø)`
agent	`49.22% <ø> (ø)`
managed	`41.84% <18.63%> (-0.46%)`	⬇️
vmproxy	`72.41% <ø> (ø)`

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

ademidoff · 2026-04-30T20:23:50Z

@angelvilardellperez Run make init && make gen to pass linters.

angelvilardellperez · 2026-05-04T07:51:28Z

@ademidoff

@angelvilardellperez Run make init && make gen to pass linters.

pushed

Add background auto-discovery that periodically scans AWS ElastiCache replication groups tagged with pmm_enable=true and registers them as Valkey services in PMM inventory. Features: - DiscoverElastiCache API endpoint for manual discovery via Swagger/UI - Background reconciler running every 5 minutes - Supports Cluster Mode Enabled (ConfigurationEndpoint) and Disabled (Primary + Reader endpoints per shard) - Filters out clusters with AUTH/ACL enabled (no credentials support) - Uses AWS "Environment" tag for PMM environment label - Auto-removes services when pmm_enable=true tag is removed - New RemoteElastiCacheNode type for inventory tracking Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

- Fix nil response on empty discovery (return empty response, not nil) - Add RemoteElastiCacheNodeType to compatibleNodeAndAgent map - Add RemoteElastiCacheNodeType to inventory-layer RemoveService cleanup - Add instance_id field to AddRemoteElastiCacheNodeParams proto - Fix misleading "unchanged" count when add failures occur Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Run make init && make gen to regenerate all derived files after ElastiCache auto-discovery changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Cover uncovered lines flagged by Codecov: - nodes.go: AddRemoteElastiCacheNode lifecycle and uniqueness - service.go: RemoveService/ListServices with ElastiCache node type - elasticache.go: region listing and engine map - elasticache_discovery.go: findManagedServices, addInstance, removeService Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

Regenerated after rebase onto latest v3 to align with updated protoc tooling and resolve generated file conflicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

CLAassistant · 2026-05-15T13:49:12Z

All committers have signed the CLA.

JiriCtvrtka · 2026-05-19T10:02:01Z

@angelvilardellperez could you run
make gen
make format
go mod tidy

to solve conflicts? Then we can proceed, thank you.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

angelvilardellperez · 2026-05-19T13:40:36Z

@angelvilardellperez could you run make gen make format go mod tidy

to solve conflicts? Then we can proceed, thank you.

pushed after running the 3 commands. Conflicts still present so i manually accepted the ones for v3

JiriCtvrtka · 2026-05-20T08:55:21Z

Yeah I can still see bunch of changed files due to formatting. Let me try to fix it.

JiriCtvrtka · 2026-05-20T09:05:59Z

@angelvilardellperez Could you check if now PR contains only your changes? If yes I can do final review. Thank you.

angelvilardellperez · 2026-05-20T09:36:19Z

@JiriCtvrtka all good

JiriCtvrtka · 2026-05-20T10:08:28Z

+	// Remove stale.
+	var removed int
+	for addr, svc := range managedByAddr {
+		if _, exists := expectedByAddr[addr]; exists {
+			continue
+		}
+		if err := d.removeService(ctx, svc); err != nil {
+			d.l.Warnf("Failed to remove %s (%s): %v", svc.ServiceName, addr, err)
+			continue
+		}
+		removed++


Region and tag lookup failures are currently treated as successful empty results. That means reconciliation can run with partial discovery data and remove existing managed services that were only missing because AWS calls failed. We should skip stale-removal unless the discovery scan completed successfully.

What do you think?

commit pushed

Region and tag-lookup failures were treated as successful empty results, so reconcile could remove managed services whose AWS calls had only failed transiently. Now propagate a scanComplete flag from checkTags through discoverRegionTagged to discoverTaggedInstances, and gate the stale-removal loop on it. Adds still proceed on partial scans since they only act on clusters we did successfully observe as tagged. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

buf treats "ElastiCache" as two words, so the ENUM_VALUE_PREFIX rule expected DISCOVER_ELASTI_CACHE_ENGINE_*. Renamed the three enum values and propagated to the regenerated pb.go (including rawDesc length prefixes), swagger JSONs, json client, and Go callers. Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

angelvilardellperez marked this pull request as ready for review April 30, 2026 07:35

angelvilardellperez requested a review from a team as a code owner April 30, 2026 07:35

angelvilardellperez requested review from JiriCtvrtka and maxkondr and removed request for a team April 30, 2026 07:35

angelvilardellperez force-pushed the pmm-autodiscover branch from e4b42b2 to 22d0be5 Compare May 4, 2026 08:01

angelvilardellperez and others added 6 commits May 5, 2026 16:15

chore: regenerate protobuf, swagger, and JSON client code

8a1a3a9

Run make init && make gen to regenerate all derived files after ElastiCache auto-discovery changes. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

fix: align struct fields in nodes_test.go for gofumpt

db23fba

Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

chore: regenerate protobuf, swagger, and JSON client code

f4f6a68

Regenerated after rebase onto latest v3 to align with updated protoc tooling and resolve generated file conflicts. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>

angelvilardellperez force-pushed the pmm-autodiscover branch from 646879d to f4f6a68 Compare May 5, 2026 14:20

barryib mentioned this pull request May 5, 2026

Auto-discovery mechanism for Valkey and selective cluster monitoring by tag #5300

Open

1 task

angelvilardellperez and others added 2 commits May 19, 2026 13:01

chore: regenerate protobuf, swagger, and JSON client code

daa9f5a

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

Merge branch 'v3' into pmm-autodiscover

99fae7f

angelvilardellperez and others added 2 commits May 20, 2026 09:45

Merge branch 'v3' into pmm-autodiscover

3f30723

Merge branch 'v3' into pmm-autodiscover

9e118c0

JiriCtvrtka added 2 commits May 20, 2026 11:03

Gen and format.

0d18ccc

Format.

b5cdbbf

JiriCtvrtka reviewed May 20, 2026

View reviewed changes

angelvilardellperez and others added 4 commits May 20, 2026 12:32

Merge branch 'v3' into pmm-autodiscover

4ea3e02

Merge branch 'v3' into pmm-autodiscover

3aba34e

JiriCtvrtka self-requested a review May 21, 2026 07:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add ElastiCache auto-discovery for Valkey/Redis services#5303

Add ElastiCache auto-discovery for Valkey/Redis services#5303
angelvilardellperez wants to merge 16 commits into
percona:v3from
angelvilardellperez:pmm-autodiscover

angelvilardellperez commented Apr 29, 2026

Uh oh!

barryib commented Apr 29, 2026

Uh oh!

codecov Bot commented Apr 30, 2026 •

edited

Loading

Uh oh!

ademidoff commented Apr 30, 2026

Uh oh!

angelvilardellperez commented May 4, 2026

Uh oh!

CLAassistant commented May 15, 2026 •

edited by it-percona-cla

Loading

Uh oh!

JiriCtvrtka commented May 19, 2026

Uh oh!

angelvilardellperez commented May 19, 2026

Uh oh!

JiriCtvrtka commented May 20, 2026

Uh oh!

JiriCtvrtka commented May 20, 2026

Uh oh!

angelvilardellperez commented May 20, 2026

Uh oh!

JiriCtvrtka May 20, 2026

Uh oh!

angelvilardellperez May 20, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

Conversation

angelvilardellperez commented Apr 29, 2026

Summary

Features

New files

Modified files

How it works

AWS permissions required

Testing

Labels applied to auto-discovered services

Uh oh!

barryib commented Apr 29, 2026

Uh oh!

codecov Bot commented Apr 30, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Codecov Report

Uh oh!

ademidoff commented Apr 30, 2026

Uh oh!

angelvilardellperez commented May 4, 2026

Uh oh!

CLAassistant commented May 15, 2026 • edited by it-percona-cla Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

JiriCtvrtka commented May 19, 2026

Uh oh!

angelvilardellperez commented May 19, 2026

Uh oh!

JiriCtvrtka commented May 20, 2026

Uh oh!

JiriCtvrtka commented May 20, 2026

Uh oh!

angelvilardellperez commented May 20, 2026

Uh oh!

JiriCtvrtka May 20, 2026

Choose a reason for hiding this comment

Uh oh!

angelvilardellperez May 20, 2026

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

6 participants

codecov Bot commented Apr 30, 2026 •

edited

Loading

CLAassistant commented May 15, 2026 •

edited by it-percona-cla

Loading