If we delete all icmp targets and update the configuration via the HUP signal, all metrics of deleted targets remain in their latest state in the export and are no longer updated.
The issue only occurs if we delete all the targets.
If we don't delete all the targets, leaving one, everything works as it should and metrics of the deleted targets are cleared correctly.
How to reproduce.
Here is the simple configuration.
conf:
refresh: 15m
nameserver_timeout: 250ms
icmp:
interval: 3s
timeout: 1s
count: 6
targets:
- name: test-1
host: 192.168.88.1
type: ICMP
- name: test-2
host: 192.168.88.2
type: ICMP
- name: test-3
host: 192.168.88.3
type: ICMP
Start the service and than get the metrics by command:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_
The output look like this:
ping_loss_percent{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 0
ping_loss_percent{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 1
ping_loss_percent{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 1
.....
.....
ping_rtt_snt_seconds{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_status{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 1
ping_status{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 0
ping_status{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_targets 3
ping_up 1
then we remove all the targets, bringing the configuration to this state:
conf:
refresh: 15m
nameserver_timeout: 250ms # Optional: DNS resolution timeout
icmp:
interval: 3s
timeout: 1s
count: 6
targets:
and send HUP signal: killall -HUP network_exporter
The service removes all targets correctly so we see console messages:
level=INFO source=signal.go:25 msg=msg !BADKEY="ReLoading config"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-1 192.168.88.1"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-2 192.168.88.2"
level=INFO source=monitor_ping.go:173 msg="Removing Target" type=ICMP func=RemoveTarget target="test-3 192.168.88.3"
then, GET metrics again:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_
ping_loss_percent{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 0
ping_loss_percent{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 1
ping_loss_percent{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 1
.....
.....
ping_rtt_snt_seconds{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_status{name="test-1",target="192.168.88.1",target_ip="192.168.88.1"} 1
ping_status{name="test-2",target="192.168.88.2",target_ip="192.168.88.2"} 0
ping_status{name="test-3",target="192.168.88.3",target_ip="192.168.88.3"} 0
ping_targets 3
ping_up 1
Metrics isn't deleted,
And counters is frozen in their last state, including incorrect state of ping_targets metric.
If we delete all icmp targets and update the configuration via the HUP signal, all metrics of deleted targets remain in their latest state in the export and are no longer updated.
The issue only occurs if we delete all the targets.
If we don't delete all the targets, leaving one, everything works as it should and metrics of the deleted targets are cleared correctly.
How to reproduce.
Here is the simple configuration.
Start the service and than get the metrics by command:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_
The output look like this:
then we remove all the targets, bringing the configuration to this state:
and send HUP signal: killall -HUP network_exporter
The service removes all targets correctly so we see console messages:
then, GET metrics again:
curl http://localhost:9427/metrics | grep -v "#" | grep ping_
Metrics isn't deleted,
And counters is frozen in their last state, including incorrect state of ping_targets metric.