PMM-15050 Hi-load performance improvements

[ademidoff]

PMM-15050

FB: SUBMODULES-4350

Summary

When PMM Server is restarted (for example during a version migration), all PMM Clients lose their long-lived gRPC Connect streams and reconnect at roughly the same instant. With the previous defaults a fleet of 800 agents would issue 800 simultaneous Grafana lookups, exhaust Grafana's request capacity, fan out into hundreds of Postgres backends, and trip Postgres' max_connections — turning the migration into a reconnect storm and a "too many clients" livelock.

This PR addresses the storm at four layers.

pmm-managed auth cache

• Longer auth cache TTL (3 s → 60 s) in AuthServer. Once an agent's credentials have been validated, the next reconnect within a minute is a free cache hit instead of another Grafana round-trip.
• Longer auth timeout (3 s → 15 s). A temporarily slow Grafana no longer immediately translates to mass 401s in nginx auth_request.
• Larger HTTP transport pool for the Grafana client: MaxIdleConns 50 → 200 and explicit MaxIdleConnsPerHost 100 (Go default is 2). Reconnect bursts no longer force a fresh TCP/TLS handshake for almost every request.

Postgres connection pressure

• Bound Grafana's Postgres pool in grafana.ini: max_open_conn = 100, max_idle_conn = 25. Grafana 11's default for max_open_conn is 0 (unlimited), so a synchronized burst of token validations can ask Grafana to open hundreds of Postgres backends and exhaust max_connections. Capping it at 100 contains the blast radius without compromising steady-state performance.
• Raise pmm-managed's Postgres pool from 5 idle / 10 open to 20 idle / 50 open. The previous limit caused DB-bound auth paths (LBAC role lookup, settings reads, agent-state writes) to queue at 10 during the burst, on the same Postgres backends Grafana was contending for.

pmm-agent reconnect

• Wider reconnect backoff jitter (±25 % → ±50 %) in the shared agent/utils/backoff package. Spreads the first retry of 800 agents across a 1 s window instead of 0.5 s. The cap (backoffMaxDelay) stays at 15 s so an agent in repeated failure does not hold up to a minute of metrics in its local buffer.

Drive-bys

• Replaces deprecated errors.Cause with errors.As in auth_server.go.
• Replaces interface{} with any in the Grafana client to clear pre-existing lint warnings in files touched here.

Notes

• The delayJitter constant lives in the shared agent/utils/backoff package, so wider jitter also applies to the slowlog reader retry and the exporter restart loop. Both are local to a single agent process; wider jitter is strictly more decorrelating there and has no downside.
• The Grafana PostgreSQL datasource (used by dashboards) is a separate connection pool not bounded by this PR. It's typically a steady-state load, not a reconnect-burst contributor; can be revisited if dashboards are seen contributing.

🤖 Generated with Claude Code

[codecov]

Codecov Report

❌ Patch coverage is 92.85714% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 42.20%. Comparing base (2396f7d) to head (c3dfc02).

Files with missing lines	Patch %	Lines
managed/services/grafana/client.go	90.00%	1 Missing ⚠️

Additional details and impacted files

@@           Coverage Diff           @@
##               v3    #5338   +/-   ##
=======================================
  Coverage   42.20%   42.20%           
=======================================
  Files         410      410           
  Lines       41995    41997    +2     
=======================================
+ Hits        17723    17725    +2     
  Misses      22488    22488           
  Partials     1784     1784           

Flag	Coverage Δ
admin	34.89% <ø> (ø)
agent	49.23% <ø> (ø)
managed	40.72% <92.85%> (+<0.01%)	⬆️
vmproxy	72.41% <ø> (ø)

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.