PMM-15050 Hi-load performance improvements

[ademidoff]

PMM-15050

SUBMODULES-4350

Summary

When PMM Server is restarted (for example during a version migration), all PMM Clients lose their long-lived gRPC Connect streams and reconnect at roughly the same instant. With the previous defaults a fleet of 800 agents would issue 800 simultaneous Grafana lookups, exhaust Grafana's request capacity, fan out into hundreds of Postgres backends, and trip Postgres' max_connections — turning the migration into a reconnect storm and a "too many clients" livelock. On top of that, each agent's vmagent holds up to 1 GiB of buffered metrics that flush to VictoriaMetrics in parallel as soon as the connection is re-established, which can swamp VM and re-feed the same loop.

This PR addresses the storm across several layers.

pmm-managed auth cache

• Longer auth cache TTL (3 s → 60 s) in AuthServer. Once an agent's credentials have been validated, the next reconnect within a minute is a free cache hit instead of another Grafana round-trip. Entries are now also age-checked on read, so a hit never serves credentials older than the TTL — previously a stale entry could survive until the next background sweep (up to ~2× the interval).
• Longer auth timeout (3 s → 15 s). A temporarily slow Grafana no longer immediately translates to mass 401s in nginx auth_request.
• Larger HTTP transport pool for the Grafana client: MaxIdleConns 50 → 100 with explicit MaxIdleConnsPerHost 100 (Go default is 2). pmm-managed only ever talks to a single Grafana host, so the per-host cap is the real bound and both are set to the same value — aligned with Grafana's max_open_conn = 100, the ceiling on auth lookups it can process in parallel. Reconnect bursts no longer force a fresh TCP connection for almost every request.

Postgres connection pressure

• Bound Grafana's Postgres pool in grafana.ini: max_open_conn = 100, max_idle_conn = 25. Grafana 11's default for max_open_conn is 0 (unlimited), so a synchronized burst of token validations can ask Grafana to open hundreds of Postgres backends and exhaust max_connections.
• Raise pmm-managed's Postgres pool from 5 idle / 10 open to 20 idle / 50 open. The previous limit caused DB-bound auth paths to queue at 10 during the burst.

VictoriaMetrics ingestion

• Make VictoriaMetrics' ingestion rate limit configurable through the VM_maxIngestionRate env var (following the existing VM_search_* / VM_promscrape_* pattern), wired to VM's --maxIngestionRate flag. The default is 0 — VM's native "no limit" — so out-of-the-box ingestion behavior is unchanged. A hard default cap was deliberately avoided: --maxIngestionRate pauses ingestion when the limit is exceeded rather than shedding load, so a fixed cap would throttle legitimate ingestion on large fleets — penalizing exactly the big deployments this work targets. Operators who need to smooth a reconnect burst can opt in by setting VM_maxIngestionRate to a value suited to their fleet; vmagent tolerates throttling natively by holding data in its 1 GiB disk queue and retrying.
• Note: this means the PR does not ship an on-by-default mitigation for the VM-swamping scenario described above — it exposes the knob for deployments that need it. An appropriately-sized default (or a soft, load-shedding limit) can be a follow-up.

pmm-agent reconnect

• Wider reconnect backoff jitter (±25 % → ±50 %) in the shared agent/utils/backoff package. Spreads the first retry of 800 agents across a 1 s window instead of 0.5 s. The cap (backoffMaxDelay) stays at 15 s so an agent in repeated failure does not hold up to a minute of metrics in its local buffer.

Drive-bys

• Replaces deprecated errors.Cause with errors.As in auth_server.go.

Notes

• The delayJitter constant lives in the shared agent/utils/backoff package, so wider jitter also applies to the slowlog reader retry and the exporter restart loop. Both are local to a single agent process; wider jitter is strictly more decorrelating there and has no downside.
• An earlier revision of this PR added singleflight around the cache miss. It was removed: each PMM Client authenticates with its own per-node service token, so cache keys are unique per agent and singleflight provides essentially no deduplication for the migration scenario.
• The Grafana PostgreSQL datasource (used by dashboards) is a separate connection pool not bounded by this PR. It's typically a steady-state load, not a reconnect-burst contributor; can be revisited if dashboards are seen contributing.
• No UI surface is affected.

[codecov]

Codecov Report

❌ Patch coverage is 75.34247% with 36 lines in your changes missing coverage. Please review.
✅ Project coverage is 44.16%. Comparing base (31318c7) to head (3dc388f).
⚠️ Report is 27 commits behind head on main.

Files with missing lines	Patch %	Lines
managed/services/management/node.go	75.32%	13 Missing and 6 partials ⚠️
managed/services/management/service.go	72.58%	13 Missing and 4 partials ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##             main    #5338      +/-   ##
==========================================
+ Coverage   43.59%   44.16%   +0.56%     
==========================================
  Files         415      416       +1     
  Lines       43134    43185      +51     
==========================================
+ Hits        18804    19071     +267     
+ Misses      22454    22231     -223     
- Partials     1876     1883       +7     

Flag	Coverage Δ
admin	34.78% <ø> (ø)
agent	50.14% <ø> (+1.09%)	⬆️
managed	43.41% <75.34%> (+0.42%)	⬆️
vmproxy	72.22% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Harness.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[JiriCtvrtka]

This comment assumes Postgres max_connections=2000, which is only true for bundled Postgres. HA/external Postgres deployments may have much lower limits, and with Grafana 100 + pmm-managed 50 per node we can exceed them. Should we point this out in our docs?

[ademidoff]

Good question! Yes, I think we need to make sure our Kubernetes distros configure the same parameter by default, then we can skip mentioning it explicitly in the docs. Otherwise we need to mention.

[maxkondr]

In general: this solution is very questionable. On one hand it increases the number of allowed incoming PG connections. On the other hand it increases the load on PG itself because there will be more requests from pmm-managed that shifts the initial problem described in ticket from pmm-managed+Grafana to Grafana+PG.

Here is the list of issues:

1. auth cache in grafana/auth_server.go doesn't hold negative responses from Grafana -> wrong credentials or DDOS-attack will always hit Grafana.

1.1. There is one common lock (mutex) on cache that creates contention lock on high load.
Good solution here would be using sharded cache with local shard's lock instead.

2. there is no protection on pmm-managed side agains "Thundering herd" problem during massive agents registration and receiving massive incoming API requests - it just tries to swallow everything it receives (by increasing the PG connections and Grafana client connections limits) and shifts the problem to backing services (Grafana, PG).

Resilient systems are built using the following approaches.
On pmm-managed side:

• introduce rate-limiter + circuit breaker + re-tries backoff with increasing jitter in Grafana client. Each retry to Grafana has a local timeout (let's say 5s) and general time for processing request (let's say 15s).
• the same as above in managed/services/agents/registry.go:register func.
• cache positive and negative responses from backing services
• during pmm-agent registration pmm-managed may calculate some jitter/coefficient that client will use later for during re-registration (and other logics like sending QAN packets). This jitter may be based on number of pmm-agents registered in DB.

On pmm-agent side the same approaches:

• for connection attempts - implement circuit breaker + re-tries backoff increasing with jitter (some sort of that is already implemented)

Regarding DB migration issue on pmm-managed side.
DB migration is part of pmm-managed component startup logic. It shall not accept incoming connections/requests from agents/API until the migration is finished. the finished migration procedure shall be included into "healthy condition" to accept incoming requests.

[ademidoff]

In general: this solution is very questionable. On one hand it increases the number of allowed incoming PG connections. On the other hand it increases the load on PG itself because there will be more requests from pmm-managed that shifts the initial problem described in ticket from pmm-managed+Grafana to Grafana+PG.

Here is the list of issues:

1. auth cache in grafana/auth_server.go doesn't hold negative responses from Grafana -> wrong credentials or DDOS-attack will always hit Grafana.

1.1. There is one common lock (mutex) on cache that creates contention lock on high load. Good solution here would be using sharded cache with local shard's lock instead.

2. there is no protection on pmm-managed side agains "Thundering herd" problem during massive agents registration and receiving massive incoming API requests - it just tries to swallow everything it receives (by increasing the PG connections and Grafana client connections limits) and shifts the problem to backing services (Grafana, PG).

Resilient systems are built using the following approaches. On pmm-managed side:

• introduce rate-limiter + circuit breaker + re-tries backoff with increasing jitter in Grafana client. Each retry to Grafana has a local timeout (let's say 5s) and general time for processing request (let's say 15s).
• the same as above in managed/services/agents/registry.go:register func.
• cache positive and negative responses from backing services
• during pmm-agent registration pmm-managed may calculate some jitter/coefficient that client will use later for during re-registration (and other logics like sending QAN packets). This jitter may be based on number of pmm-agents registered in DB.

On pmm-agent side the same approaches:

• for connection attempts - implement circuit breaker + re-tries backoff increasing with jitter (some sort of that is already implemented)

Regarding DB migration issue on pmm-managed side. DB migration is part of pmm-managed component startup logic. It shall not accept incoming connections/requests from agents/API until the migration is finished. the finished migration procedure shall be included into "healthy condition" to accept incoming requests.

I'm not sure we can go ahead with such massive refactoring given that it's very difficult to reliably test the changes. I think we should consider the refactoring in a series of separate tickets.