Skip to content

Prometheus Metrics remains if all targets are deleted #80

@archer-v

Description

@archer-v

If we delete all icmp targets and update the configuration via the HUP signal, all metrics of deleted targets remain in their latest state in the export and are no longer updated.

The issue only occurs if we delete all the targets.
If we don't delete all the targets, leaving one, everything works as it should and metrics of the deleted targets are cleared correctly.

How to reproduce.

Here is the simple configuration.

conf:
  refresh: 15m
  nameserver_timeout: 250ms 

icmp:
  interval: 3s
  timeout: 1s
  count: 6

targets:
  - name: test-1
    host: 192.168.88.1
    type: ICMP
  - name: test-2
    host: 192.168.88.2
    type: ICMP
  - name: test-3
    host: 192.168.88.3
    type: ICMP

Start the service and than get the metrics by command:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_

The output look like this:

ping_loss_percent{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 0
ping_loss_percent{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 1
ping_loss_percent{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 1
.....
.....
ping_rtt_snt_seconds{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_status{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 1
ping_status{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 0
ping_status{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_targets 3
ping_up 1

then we remove all the targets, bringing the configuration to this state:

conf:
  refresh: 15m
  nameserver_timeout: 250ms # Optional: DNS resolution timeout

icmp:
  interval: 3s
  timeout: 1s
  count: 6

targets:

and send HUP signal: killall -HUP network_exporter

The service removes all targets correctly so we see console messages:

level=INFO source=signal.go:25 msg=msg !BADKEY="ReLoading config"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-1 192.168.88.1"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-2 192.168.88.2"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-3 192.168.88.3"

then, GET metrics again:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_

ping_loss_percent{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 0
ping_loss_percent{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 1
ping_loss_percent{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 1
.....
.....
ping_rtt_snt_seconds{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_status{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 1
ping_status{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 0
ping_status{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_targets 3
ping_up 1

Metrics isn't deleted,
And counters is frozen in their last state, including incorrect state of ping_targets metric.

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions