Any chance for replication factor > 1 support?

[nbrownus]

So happy to see someone has picked up the project. Wondering if supporting replication is on the roadmap.

[azhiltsov]

We currently does not have plans for it as we have stopped using replication factor quite a while ago. We are running setup where carbon-c-relay create multiple copies of the same metrics. This gives us more predictability for their placement on a stores, and more predictable failure scenario when you have more than one store failure. Another pros will be back-filling of data after a failure. With replication factor = 1 and equally sized rings it easy to find the source host for back-fill.

[aptituz]

@azhiltsov That sounds interesting. Could you elaborate a bit on your setup? Do you have multiple carbon clusters for that copy-based "replication"?

[Civil]

https://fosdem.org/2017/schedule/event/graphite_at_scale/ - there are more information there. But TLDR:

1. With replication factor 1 and duplication on Relay level you have 1 to 1 mapping of the hosts. You always will be able to easily find the metric.
2. It's basically tradeoff of loosing some data and probability of loosing it. With RF >1 - you will always loose some data in case of 2 servers failure. It will be a small portion of data, but still something. With RF=1 but 2 clusters having same data - probability to loose data will be lower (conditional probability), but if it will happen - amount of lost data will be higher.

You can think about that as about Raid5 vs. Raid10, with an exception that if you'll loose n+1 server when you have RF= n you won't loose all the data, but only portion of it.

[azhiltsov]

@aptituz our carbon-c-relay config snippet would look like this:

cluster carbon_a
    jump_fnv1a_ch replication 1
        10.19.19.24:2003=000
        10.19.20.34:2003=001
   ;
cluster carbon_b
    jump_fnv1a_ch replication 1
        10.18.19.4:2003=000
        10.18.20.4:2003=001
   ;
match ^carbon\.
    send to
        carbon_a
        carbon_b
    stop;

the 'match' section above is what actually creates two replicas.

[aptituz]

And how is this handled on the other side of the stack? Is graphite-web/carbonapi/carbonzipper or whatever is used to query the data just configured to use the hosts of both clusters?

[Civil]

You just query all the servers from a carbonzipper.

[aptituz]

Thanks! :)