v0.72.0

Update note 1: vmalert: rule ConfigMaps now store gzip-compressed content in binaryData and an init container decompresses them before VMAlert starts. Existing VMAlert pods will be rolled out once during this upgrade.

• FEATURE: vmoperator: add validating webhooks for Prometheus Operator CRDs (ServiceMonitor, PodMonitor, PrometheusRule, Probe, ScrapeConfig, AlertmanagerConfig). Each object is converted to its VM equivalent and validated when webhooks are enabled. See #2270.

• FEATURE: vmscrapeconfig: add support for consulAgentSDConfigs, dockerSDConfigs, dockerSwarmSDConfigs, marathonSDConfigs, and yandexCloudSDConfigs service discovery types, bringing VMScrapeConfig to full parity with VictoriaMetrics sd_configs. See #2265.

• FEATURE: vmalert: rule ConfigMaps now store gzip-compressed rule files in binaryData, allowing larger rule sets within Kubernetes object size limits. An init container decompresses the rules before VMAlert starts.

• FEATURE: vmscrapeconfig: add support for consulAgentSDConfigs, dockerSDConfigs, dockerSwarmSDConfigs, marathonSDConfigs, and yandexCloudSDConfigs service discovery types, bringing VMScrapeConfig to full parity with VictoriaMetrics sd_configs. See #2265.

• FEATURE: vmoperator: add victoriametrics_app=true label to all metrics scraped by the operator. See #2261.

• BUGFIX: vmoperator: skip reconciliation only for CRs with genuine spec parse errors; CRs whose ParsingSpecError is caused solely by unknown fields (e.g. after an operator downgrade) are now reconciled normally instead of being silently skipped.

• BUGFIX: vmoperator: fix potential deadlock in operator_object_status metrics collector when the number of tracked objects exceeds 250. The Collect method previously held a mutex while sending to the prometheus channel, which could deadlock if the channel was full and another goroutine was waiting on the same mutex. See #2239.

• BUGFIX: config-reloader: fix missed reload for watched files whose names contain .. (e.g. rules..yaml). Previously any path containing .. was silently skipped; now only Kubernetes synthetic entries whose basename starts with .. (e.g. ..data) are ignored. See #2253.

• BUGFIX: vmoperator: fix potential deadlock in operator_object_status metrics collector when the number of tracked objects exceeds 250. The Collect method previously held a mutex while sending to the prometheus channel, which could deadlock if the channel was full and another goroutine was waiting on the same mutex. See #2239.

v0.71.0

Update note 1: the new default preStop hook causes a rolling update of all applicable pods on operator upgrade (on Kubernetes >= 1.29). To avoid this, set VM_ENABLE_DEFAULT_PRESTOP_HOOK=false on the operator before upgrading. Once the upgrade is complete, you can re-enable it by removing the override (or setting it to true) to roll out the hook at a time of your choosing. Alternatively, disable the hook per resource by setting spec.preStopSleepSeconds: 0.

• Dependency: vmoperator: Updated default versions for VM apps to v1.145.0 version

• FEATURE: vmoperator: added VM_COMMON_LABELS and VM_COMMON_ANNOTATIONS environment variables to apply common labels/annotations to all Kubernetes resources managed by the operator. These cannot override labels/annotations already set by the operator or via spec.managedMetadata. This also ensures HTTPRoutes and PVCs include ManagedMetadata labels and annotations

• FEATURE: vmoperator: support enableServiceLinks property in all CRs. See #2194.

• FEATURE: vmalertmanagerconfig: add url_file and alert_source_token_file fields to IncidentioConfig, as file-based alternatives to url and alert_source_token. See #2222.

• FEATURE: vmoperator: add status metrics for objects managed by each controller. See #2238.

• FEATURE: vmsingle: added spec.downsampling for structured downsampling configuration and spec.retentionFilters for structured retention filters configuration. Both require an enterprise license.

• FEATURE: vmcluster: added spec.downsampling for structured downsampling configuration (applied to both vmselect and vmstorage) and spec.vmstorage.retentionFilters for structured retention filters configuration. Both require an enterprise license.

• FEATURE: vmoperator: add default preStop lifecycle hook (15s sleep) to applicable component pods to prevent traffic loss during pod termination. Requires Kubernetes >= 1.29. Storage components (VMStorage, VLStorage, VTStorage) are excluded since their clients handle retries. The sleep duration is configurable via spec.preStopSleepSeconds on any CR; set to 0 to disable. See #1834.

• FEATURE: vmanomaly: add missing configuration parameters for vmanomaly config parity: spec.reader.offset and per-query offset, spec.writer.connectionRetryAttempts, spec.server.useReaderConnectionSettings, settings.logger_levels, and the exact/infer_every parameters of the backtesting scheduler. Previously these keys were rejected by strict config parsing.

• FEATURE: vmcluster: add spec.discovery field with enabled, interval and filter properties to configure automatic vmstorage node discovery for vminsert and vmselect. The field can be set globally or overridden per component via spec.vminsert.discovery and spec.vmselect.discovery. This is an enterprise feature and requires a valid license key. See automatic vmstorage discovery.

• BUGFIX: vmoperator: update status currentRevision and currentReplicas for StatefulSet with OnDelete update strategy. See #1242.

• BUGFIX: config-reloader: fix configreloader_last_reload_success_timestamp_seconds metric to report time in seconds instead of milliseconds.

• BUGFIX: vmoperator: enable strict CR spec unmarshalling when creating objects. See #2882.

• BUGFIX: vmoperator: ignore NotFound errors, that may occur during attempt to update status on a missing resource.

• BUGFIX: vmagent, vmanomaly: fix VPA scale subresource lookup failure when spec.shardCount is unset by always reporting at least 1 in status.shards. See #2229.

• BUGFIX: vmagent: fix HPA targeting the underlying Deployment/StatefulSet (pod replicas) instead of the VMAgent CR scale subresource (spec.shardCount); HPA now correctly scales the number of shards. See #2229.

• BUGFIX: vmanomaly: emit the OnlineQuantileModel smoothing parameter under its correct key global_smoothing instead of the unrecognized global_smooth, which vmanomaly silently ignored.

• BUGFIX: vmanomaly: pass the configured TLS CA bundle to the reader, writer and monitoring clients. Previously the CA was mounted as a volume but dropped during config generation, so a tlsConfig with only a CA produced no verify_tls reference to it; insecureSkipVerify is now also propagated correctly.

• BUGFIX: vmanomaly: serialize the tz timezone of spec.reader, of each query, and of the periodic scheduler as a string. It was serialized to {} and could not be parsed from a string, so any configuration that set tz failed to reconcile.

• BUGFIX: vmanomaly: fix the AutoTunedModel trials parameter spelling n_trials, previously it was rendered as n_trails which was silently ignored by vmanomaly.

• BUGFIX: vmanomaly: preserve an explicit 0.0 for the anomaly_score_outside_data_range parameter (both settings and model level), which was dropped by the underlying float field and silently reverted to the vmanomaly default.

• BUGFIX: vmanomaly: validate online models' decay against the (0, 1] range while still allowing it to be omitted.

• BUGFIX: vmanomaly: accept a float value (for example 0.01) for the isolation forest contamination parameter in addition to the string auto.

• BUGFIX: vmanomaly: omit the OnlineQuantileModel min_subseason key when it is unset instead of emitting an empty string.

• BUGFIX: vmanomaly: remove the artificial upper bound on spec.server.maxConcurrentTasks; vmanomaly accepts any positive integer.

v0.68.5

Update note: -eula flag is not set by default anymore for VMBackup and VMRestore. To avoid VMCluster/VMSingle rollouts set spec.vmstorage.vmBackup.acceptEula: true for VMCluster and spec.vmBackup.acceptEula: true for VMSingle and replace it with spec.license during VMSingle/VMCluster upgrade.

• Dependency: vmoperator: Updated default versions for VM apps to v1.144.0 version

• SECURITY: upgrade Go builder from Go1.25.8 to Go1.25.10. See the list of issues addressed in Go1.25.10.

• BUGFIX: vmagent: use volume from spec.volumes as persistent queue volume if its name is persistent-queue-data, previously emptyDir was mounted. See #1677.

• BUGFIX: vmcluster: use volume from spec.vmstorage.volumes and spec.vmselect.volumes as data and cache volumes if its name is vmstorage-db and vmselect-cachedir respectively. See #784.

• BUGFIX: vmoperator: Improve reconcile error handling for Prometheus and VictoriaMetrics controllers.

• BUGFIX: vmoperator: Add acceptEula support for VMBackup/VMRestore.

• BUGFIX: vmdistributed: change default load balancing policy for write requests from first_available to least_loaded. This should allow to evenly distribute write load across all VMAgents.

• BUGFIX: VMCluster, VTCluster and VLCluster: fixed infinite non-default additional service recreation, when requestsLoadBalancer.enabled: true

• BUGFIX: vmoperator: retry reconcile errors, that may lead to expanding state, before resource could hang in expanding state.

• BUGFIX: vmdistributed: expose VMClusterSpec parsing error in status, previously it was just swallowed and led to infinite reconciles. See #2113.

• BUGFIX: vmanomaly and vmagent: Fix incorrect scaling logs for sharded vmagent and vmanomaly.

v0.70.1

• FEATURE: vmauth: support HPA for requests load balancer.

• BUGFIX: vmagent: skip replica count update when VMAgent is in stateful mode and HPA is enabled. See #2190.

• BUGFIX: vmalertmanagerconfig: Remove prefix from empty subroute receiver. See #2185.

• BUGFIX: vmoperator: updated OLM configuration to fix OpenShift catalog publishing. See #2195.

v0.70.0

• SECURITY: upgrade Go builder from Go1.26.2 to Go1.26.3. See the list of issues addressed in Go1.26.3.

• Dependency: vmoperator: Updated default versions for VM apps to [v1.143.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.
  143.0) version

• Dependency: vmoperator: Updated default versions for VM apps to [v1.142.0](https://github.com/VictoriaMetrics/VictoriaMetrics/releases/tag/v1.
  142.0) version

• FEATURE: vmauth: previously VMAuth could read configuration only from predefined locations; now VMAuth supports arbitrary filesystem access configuration, allowing users to reference required files directly and reducing configuration workarounds. See #899.

• FEATURE: vmuser: support VMAnomaly CRD in VMUser targetRefs. See [#2141](#2
  141).

• FEATURE: vmdistributed: introduce spec.zones[*].trafficMode property, which allows disable read, write or whole traffic to a zone. See #1995.

• FEATURE: vmagent: support per remote write queues configuration. See [#2138](https://github.com/VictoriaMetrics/operator/iss
  ues/2138).

• BUGFIX: converter: disable all prometheus controllers if CRD group was not found. See [#2838](https
  ://github.com/VictoriaMetrics/helm-charts/issues/2838).

• BUGFIX: vmdistributed: change default load balancing policy for write requests from first_available to least_loaded . This should allow to evenly distribute write load across all VMAgents.

• BUGFIX: vmalertmanagerconfig: fix previously ignored negative values in VMAlertmanagerConfig. See [#2132](https
  ://github.com//issues/2132).

• BUGFIX: vmalertmanager: fixed ignored alertmanager template if it has no discovered VMAlertmanagerconfig CRs or traci
  ng config defined. See #2121.

• BUGFIX: VMCluster, VTCluster and [VLCluster](https://doc
  s.victoriametrics.com/operator/resources/vlcluster/): fixed infinite non-default additional service recreation, when requestsLoadBalancer.enabled: true

• BUGFIX: vmauth: fixed case, when target_path_suffix can be appended multiple times.

• BUGFIX: vmoperator: retry reconcile errors, that may lead to expanding state, before resource could hang in expanding state.

• BUGFIX: vmcluster, vlcluster and [vtcluster](https://doc
  s.victoriametrics.com/operator/resources/vtcluster/): when storage HPA was enabled, generated -storageNode flags could become incorrect after scaling, which could break expected routing to
  storage nodes; now the operator derives storage node count from the current StatefulSet state so generated flags stay correct during HPA-driven scaling. See [#2117](https://github.com/Victo
  riaMetrics/operator/issues/2117).

• BUGFIX: vmdistributed: expose VMClusterSpec parsing error in status, previously it was just swallowed and led to infin
  ite reconciles. See #2113.

v0.69.0

• Dependency: vmoperator: Updated default versions for VM apps to v1.140.0 version

• Dependency: vmoperator: Updated default versions for VM apps to v1.139.0 version
  Update note 1: deprecated env variables for Prometheus CRs conversion VM_ENABLEDPROMETHEUSCONVERTER_PODMONITOR, VM_ENABLEDPROMETHEUSCONVERTER_SERVICESCRAPE, VM_ENABLEDPROMETHEUSCONVERTER_PROMETHEUSRULE, VM_ENABLEDPROMETHEUSCONVERTER_PROBE, VM_ENABLEDPROMETHEUSCONVERTER_SCRAPECONFIG. Use -controller.disableReconcileFor command-line flag with comma-separated list of controller names, that should be disabled.
  Update note 2: removed operator_prometheus_converter_watch_events_total metric since migration of Prometheus object watchers to controllers made this counter obsolete.
  Update note 3: made controller.prometheusCRD.resyncPeriod command line flag noop, which was relevant to Prometheus object watchers.
  Update note 4: -eula flag is not set by default anymore for VMBackup and VMRestore. To avoid VMCluster/VMSingle rollouts set spec.vmstorage.vmBackup.acceptEula: true for VMCluster and spec.vmBackup.acceptEula: true" for VMSingle and replace it with spec.license during VMSingle/VMCluster upgrade.
  Update note 5: new flag was introduced, so VMAnomaly pods will be recreated. VMAnomaly version should be at least 1.25.
  Update note 6: fixed port name collision in VMSingle and VMCluster, when VMBackup is enabled. To avoid VMSingle/VMCluster pods recreation set VM_USE_OLD_BACKUP_RESTORE_PORT_NAMES environment variable to true.
  Update note 7: the default VMDistributed unauthorizedUserAccessSpec was removed to allow users to decide whether access should be anonymous or authorized. Please check documentation with examples for unauthorized and authorized access and update your CRs accordingly.

• Dependency: vmoperator: Updated default versions for VM apps to v1.139.0 version

• Dependency: vmoperator: Updated default versions for VL apps to v1.50.0.

• FEATURE: helm-converter: new CLI tool that helps with migration from Helm charts to their corresponding Operator Custom Resources (CRs).

• FEATURE: vmsingle: VMSingle reuses vmagent implementation to allow scraping and relabelling. See #1694

• FEATURE: vmoperator: perform statefulset pods deletion instead of eviction when maxUnavailable set to 100%, which is important for minimum downtime strategy. See #1706.

• FEATURE: vmuser: support referencing multiple targets of the same kind at targetRefs[*].crd.objects.

• FEATURE: vmoperator: prettify reconcile diff in logs, now diff objects show only changed JSON fields.

• FEATURE: VLCluster, VLSingle, VLAgent, VTCluster, VTSingle, VMCluster, VMAgent, VMAnomaly, VMAlert, VMAlertmanager, VMAuth: add spec.componentVersion as an alternative to spec.clusterVersion. This field also available in all objects deploying pods. See this #1949 issue for details.

• FEATURE: vmanomaly: add support for settings.retention configuration (ttl and check_interval) in configRawYaml and configSecret. See these docs for details.

• FEATURE: vmuser: support JWT-based auth.

• FEATURE: vmagent: support HPA in VMAgent CR and in VMAgent, which is a part of VMDistributed. See #1961.

• FEATURE: vmagent: VMAgent CRs running in statefulSet mode, including VMAgent components in VMDistributed, now support configuring rolling update strategy behavior. See #1987.

• FEATURE: vmagent: VMAgent CRs running in DaemonSet mode now support configuring rolling update strategy behavior.

• FEATURE: vmoperator: Dry-run mode. See #1832.

• FEATURE: vmanomaly: introduce VMAnomalyConfig CRD to enable dynamic configuration and hot-reload support starting from VMAnomaly version 1.25.0.

• FEATURE: vmalertmanager: introduce arbitrary fs access feature for VMAlertmanager. See #899

• FEATURE: vmalertmanagerconfig: add update_message field to SlackConfig. This allows alertmanager to edit the original Slack message in-place when alert status changes instead of sending a new one. Requires alertmanager v0.32.0+. See #2064.

• BUGFIX: vmbackupmanager: remove deprecated -eula flag from vmbackupmanager and vmrestore container args. See #1319.

• BUGFIX: vmoperator: VMPodScrape for VLAgent and VMAgent now uses the correct port; previously it used the wrong port and could cause scrape failures. See #1887.

• BUGFIX: vmdistributed: updated VMAuth config consolidating all VMSelects into a single read and all VMClusters into a single write backend

• BUGFIX: vmdistributed: fix PVC being owned by StatefulSet and top-level object simultaneously. See #1845.

• BUGFIX: vmoperator: remove unneeded finalizer from core K8s resources. See #835.

• BUGFIX: vmdistributed: remove finalizers from VMServiceScrape and VMPodScrape objects, and keep finalizers on VMAgent, VMCluster, and VMAuth when DeletionTimestamp is not empty.

• BUGFIX: vmsingle and vmagent: previously, ingest-only mode could still mount scrape configuration secrets when relabeling or stream aggregation was configured, which caused unexpected secret mounts and RBAC-related failures; now these secrets are not mounted in ingest-only mode, so deployments start with the expected minimal permissions and avoid related runtime errors. See #1828.

• BUGFIX: vmoperator: recreate STS if immutable fields changed.

• BUGFIX: vmoperator: wait for STS deletion in case of recreation without throwing an error.

• BUGFIX: vmdistributed: ignore VMAuth update/delete operations if controller is disabled.

• BUGFIX: vmalertmanager: fixed ignored tracing config, when no alertmanagerconfig CRs collected. See #1983.

• BUGFIX: vmagent: apply scrape class relabellings before job ones. See #1997.

• BUGFIX: vmanomaly and vmagent: render %SHARD_NUM% placeholder when shard count is greater than 0. See [#2001](https://github.com/VictoriaMetrics/operator/i...

v0.68.4

• FEATURE: vmagent: introduce statefulRollingUpdateStrategyBehavior to allow managing VMAgent update strategy in a statefulMode. See #1987.

• FEATURE: vmoperator: Dry-run mode. See #1832.

• BUGFIX: vmoperator: wait till PVC resize finished. See #1970.

• BUGFIX: vmalertmanager: fixed ignored tracing config, when no alertmanagerconfig CRs collected. See #1983.

• BUGFIX: vmagent: apply scrape class relabellings before job ones. See #1997.

• BUGFIX: vmanomaly and vmagent: render %SHARD_NUM% placeholder when shard count is greater than 0. See #2001.

• BUGFIX: vlcluster and vtcluster: do not ignore ExtraStorageNodes for select, when default storage is disabled. See #1910.

• BUGFIX: vmdistributed: use default stub, when no VMAuth backends are available

v0.68.3

• FEATURE: vmoperator: prettify reconcile diff in logs, now diff objects show only changed JSON fields.

• FEATURE: vmoperator: introduce VM_LOOPBACK env variable, which allows to override default loopback address, which is either localhost if VM _ENABLETCP6=true or 127.0.0.1 in other cases. Note: this change may cause component rollouts when 127.0.0.1 is used as the loopback address. Make sure to set this env var before upgrading
  .

• BUGFIX: vmoperator: recreate STS if immutable fields changed.

• BUGFIX: vmoperator: pod recreation isn't lost after triggerring StatefulSet delete.

• BUGFIX: vmdistributed: updated VMAuth config consolidating all VMSelects into a single read and all VMAgents into a si
  ngle write backend.

• BUGFIX: vmoperator: fix finalizer cleanup for VLAgent, VLogs, VMAgent, and VMSingle to target the correct resources.

• BUGFIX: config-reloader: log-format option supported as fallback for loggerFormat. See [#1954](https://gi
  thub.com//issues/1954).

• SECURITY: upgrade Go builder from Go1.25.7 to Go1.25.8. See the list of issues addressed in Go1.25.8.

v0.68.3-rc0

• FEATURE: vmoperator: prettify reconcile diff in logs, now diff objects show only changed JSON fields.

• FEATURE: vmoperator: introduce VM_LOOPBACK env variable, which allows to override default loopback address, which is either localhost if VM _ENABLETCP6=true or 127.0.0.1 in other cases. Note: this change may cause component rollouts when 127.0.0.1 is used as the loopback address. Make sure to set this env var before upgrading
  .

• BUGFIX: vmoperator: recreate STS if immutable fields changed.

• BUGFIX: vmoperator: pod recreation isn't lost after triggerring StatefulSet delete.

• BUGFIX: vmdistributed: updated VMAuth config consolidating all VMSelects into a single read and all VMAgents into a si
  ngle write backend.

• BUGFIX: vmoperator: fix finalizer cleanup for VLAgent, VLogs, VMAgent, and VMSingle to target the correct resources.

• BUGFIX: config-reloader: log-format option supported as fallback for loggerFormat. See [#1954](https://gi
  thub.com//issues/1954).

• SECURITY: upgrade Go builder from Go1.25.7 to Go1.25.8. See the list of issues addressed in Go1.25.8.

v0.68.2

• Dependency: vmoperator: Updated default versions for VL apps to v1.47.0.

• BUGFIX: vmoperator: perform statefulset pods deletion instead of eviction when maxUnavailable set to 100%, which is important for minimum downtime strategy. See #1706.

• BUGFIX: vmoperator: VMPodScrape for VLAgent and VMAgent now uses the correct port; previously it used the wrong port and could cause scrape failures. See #1887.

• BUGFIX: vmdistributed: fix PVC being owned by StatefulSet and top-level object simultaenously. See #1845.

• BUGFIX: vmoperator: remove unneeded finalizer from core K8s resources. See #835.

• BUGFIX: vmdistributed: remove finalizers from VMServiceScrape and VMPodScrape objects, and keep finalizers on VMAgent, VMCluster, and VMAuth when DeletionTimestamp is not empty.

• BUGFIX: vmsingle and vmagent: previously, ingest-only mode could still mount scrape configuration secrets when relabeling or stream aggregation was configured, which caused unexpected secret mounts and RBAC-related failures; now these secrets are not mounted in ingest-only mode, so deployments start with the expected minimal permissions and avoid related runtime errors. See #1828.

• BUGFIX: vmoperator: do not recreate STS if VCT size was increased and recreate in other cases.

• BUGFIX: vmoperator: wait for STS deletion in case of recreation without throwing an error.