feature: multiple address support for --statsd.relay.address

[howardyoo]

Problem

The current --statsd.relay.address is a handy feature to forward received statsd metrics to another statsd compliant listener. However, in current implementation, you can only set a single target for relay address, and sometimes having multiple relay addresses might be necessary to send it to more than one destinations.

Solution

In the current --statsd.relay.address, provide an option to have a multiple addresses delimited by ; character, so that when provided something like this:

--statsd.relay.address=host_1:1527;host_2:3362;host_3:4523

StatsD exporter would forward the received statsd data to host_1, host_2, and host_3 on their respective ports. Implement the forward mechanism such that submitting the statsd metrics to each destination will be asynchronous and independent to each other, so that even one of the targets would fail, timeout, or not available, other viable targets would still receive the statsd metrics.

[matthiasr]

I think this is reasonable, although we should not use ; – the flag libray can handle repeatable flags. With this we can accumulate the provided names into an array (instead of the single receiver we support today), and then send to each of them.

To consider for the implementation:

• What happens when one of the targets is slow or failing?
• Should we distinguish between the different targets in metrics? How?

[howardyoo]

I think this is reasonable, although we should not use ; – the flag libray can handle repeatable flags. With this we can accumulate the provided names into an array (instead of the single receiver we support today), and then send to each of them.

I see! Didn't know about this, so yes, looks like we can have repeated flags to have support for multiple relay addresses.

To consider for the implementation:

• What happens when one of the targets is slow or failing?
• Should we distinguish between the different targets in metrics? How?

• I would envision these multiple targets would be multi-threaded, so that one slow running forwarding may not affect the other threads. We should definitely give out an error messages in case one of the relay target has failed or timed out (definitely need time-outs to fail something if transmission is taking too long).
• I don't suppose we should distinguish between different targets in metrics. I don't see any need for doing that being included in the metrics.