HATracker: Add a local cache warmup on startup

[SungJin1212]

This PR introduces local cache warmup logic on HA tracker startup that fetches all keys from the KV store and warms the local cache.

Previously, whenever the Distributor started, it suffered from initial cold cache misses. Since the local map was empty, the HATracker treated every incoming request as a new entry. This caused unnecessary CAS operations to the KV store even for existing valid keys.

Which issue(s) this PR fixes:
Fixes #

Checklist

• Tests updated
• Documentation added
• CHANGELOG.md updated - the order of entries should be [CHANGE], [FEATURE], [ENHANCEMENT], [BUGFIX]

[friedrichg]

Do you see distributors in clusters with large number of pairs using more memory after this change? Do we need a flag for this?

[SungJin1212]

@friedrichg
I added a benchmark for syncKVStoreToLocalMap under various key size.
I ran a benchmark and the results indicate that the overhead is negligible even at a very large scale (10000 keys, about 18MB).

goos: darwin
goarch: arm64
pkg: github.com/cortexproject/cortex/pkg/ha
cpu: Apple M4 Max
BenchmarkHATracker_syncKVStoreToLocalMap
BenchmarkHATracker_syncKVStoreToLocalMap/keys=100
BenchmarkHATracker_syncKVStoreToLocalMap/keys=100-14         	   10740	    113366 ns/op	  203197 B/op	    2532 allocs/op
BenchmarkHATracker_syncKVStoreToLocalMap/keys=1000
BenchmarkHATracker_syncKVStoreToLocalMap/keys=1000-14        	    1174	    973137 ns/op	 1834239 B/op	   24289 allocs/op
BenchmarkHATracker_syncKVStoreToLocalMap/keys=10000
BenchmarkHATracker_syncKVStoreToLocalMap/keys=10000-14       	     130	   9065987 ns/op	18199241 B/op	  240955 allocs/op
PASS

[friedrichg]

@SungJin1212 Thanks, this is great work.

[yeya24]

I am a bit doubt of how much this cache warmup would help.
I think at least we never see this issue in production. If we think it is good to have I prefer to put this behind a feature flag and disable by default.

I am a bit worried that during a big fleet scale up distributors ended up sending too many requests to KV stores just to warmup cache and causing much bigger impact.

[SungJin1212]

@yeya24
I've put the warmup logic behind the flag so that users can opt-in only if they need.

[friedrichg]

I think this flag might also prove useful for #7220.

At least bring us closer

[friedrichg]

@SungJin1212 maybe also mark it as experimental and include the flag in https://github.com/cortexproject/cortex/blob/master/docs/configuration/v1-guarantees.md

[SungJin1212]

@friedrichg
I added it, thanks!