gocached, cmd/gocached: add storage tiering and a separate SQLite dir

Our AWS cache instance stores everything on its ephemeral NVMe disk, so
the whole cache is lost on every reboot and its size is capped by the
local disk. Let the blob directory live on a big slow filesystem (an S3
Files NFS mount) as the source of truth, with a new optional --hot-dir
flag naming a fast local tier (NVMe) holding a bounded copy of recently
used blobs, capped by --hot-capacity-gb. The new --sqlite-dir flag lets
the metadata database live on its own volume (EBS), defaulting to the
blob directory as before.

PUTs write through to both tiers in one pass; a hot tier failure never
fails the PUT since a later GET re-promotes. GETs try the hot tier
first and asynchronously promote cold hits into it. The hot tier is
tracked by an in-memory LRU index seeded from an mtime-ordered scan at
startup, evicting least recently used files down to 95% of capacity;
hot files are disposable copies, so eviction and reboot loss are always
safe. When tiering is not enabled, behavior is unchanged.

Updates tailscale/corp#44699

Signed-off-by: Brad Fitzpatrick <bradfitz@tailscale.com>
Change-Id: I298711f6ee38efa15304aad7295736197c3de084

Author: bradfitz

cmd/gocached/gocached.go

@@ -22,6 +22,9 @@ import (
 
 var (
 	dir         = flag.String("cache-dir", "", "cache directory, if empty defaults to <UserCacheDir>/gocached")
+	sqliteDir   = flag.String("sqlite-dir", "", "directory for the SQLite metadata database; if empty, defaults to the cache directory")
+	hotDir      = flag.String("hot-dir", "", "if non-empty, enable storage tiering with this directory as a fast tier (e.g. local NVMe) holding a bounded copy of recently used blobs; the cache directory remains the source of truth")
+	hotCapacity = flag.Int("hot-capacity-gb", 600, "maximum size of the hot tier directory in GiB; only used with --hot-dir")
 	verbose     = flag.Bool("verbose", false, "be verbose")
 	listen      = flag.String("listen", ":31364", "listen address for the build-facing HTTP server")
 	debugListen = flag.String("debug-listen", "", "if non-empty, listen address for the debug HTTP server (pprof, metrics, etc)")
@@ -65,12 +68,23 @@ func main() {
 
 	opts := []gocached.ServerOption{
 		gocached.WithDir(*dir),
+		gocached.WithSQLiteDir(*sqliteDir),
 		gocached.WithVerbose(*verbose),
 		gocached.WithMaxSize(int64(*maxSize) << 30),
 		gocached.WithMaxAge(time.Duration(*maxAge) * 24 * time.Hour),
 		gocached.WithShardPrefixLen(*shardPrefixLen),
 	}
 
+	if *hotDir != "" {
+		if *hotCapacity <= 0 {
+			log.Fatal("--hot-capacity-gb must be positive when --hot-dir is set")
+		}
+		opts = append(opts,
+			gocached.WithHotDir(*hotDir),
+			gocached.WithHotCapacity(int64(*hotCapacity)<<30),
+		)
+	}
+
 	if *jwtIssuer != "" {
 		if len(jwtClaims) == 0 {
 			log.Fatal("must specify --jwt-claim at least once when --jwt-issuer is set")