feat: add CloudTrail S3 log source

[adityathebe]

Add support for reading CloudTrail events from existing S3 log buckets instead of the LookupEvents API.

The S3 source lists CloudTrail date prefixes, reads gzipped log files, and converts records into the existing CloudTrail processing path for changes and AssumeRole access logs.

Progress is tracked with an in-memory S3 object cursor per bucket/region/prefix, so repeated runs in the same process skip already-processed log files without adding scrape-summary metadata.

[github-actions]

Benchstat

Base: 6ab384739b1c80dc9ab37b7804712e096fe2f7a0
Head: 1c47898de3eb765d750230472420eba36e811a67

✅ No significant performance changes detected

Full benchstat output

goos: linux
goarch: amd64
pkg: github.com/flanksource/config-db/bench
cpu: AMD EPYC 9V74 80-Core Processor                
                                         │ bench-base.txt │           bench-head.txt           │
                                         │     sec/op     │    sec/op     vs base              │
LocationFilter/smallEnv-4                    18.60µ ± 20%   21.19µ ± 15%       ~ (p=0.937 n=6)
LocationFilter/largeEnv-4                    45.68µ ± 12%   45.71µ ±  1%       ~ (p=0.485 n=6)
RunTemplateBool/smallEnv-4                   10.60µ ±  1%   10.57µ ±  1%       ~ (p=0.258 n=6)
RunTemplateBool/largeEnv-4                   19.68µ ±  1%   19.67µ ±  1%       ~ (p=0.589 n=6)
BenchSaveResultsSeed/N=1000-4                 3.607 ± 11%    3.594 ± 11%       ~ (p=0.937 n=6)
BenchSaveResultsUpdateUnchanged/N=1000-4      2.966 ±  1%    2.977 ±  1%       ~ (p=0.180 n=6)
BenchSaveResultsUpdateChanged/N=1000-4        7.049 ±  3%    7.079 ±  3%       ~ (p=0.240 n=6)
geomean                                      3.885m         3.959m        +1.90%

                                         │ bench-base.txt │            bench-head.txt            │
                                         │      B/op      │     B/op      vs base                │
LocationFilter/smallEnv-4                    14.97Ki ± 0%   14.97Ki ± 0%       ~ (p=1.000 n=6) ¹
LocationFilter/largeEnv-4                    20.38Ki ± 0%   20.38Ki ± 0%       ~ (p=1.000 n=6)
RunTemplateBool/smallEnv-4                   8.359Ki ± 0%   8.359Ki ± 0%       ~ (p=1.000 n=6) ¹
RunTemplateBool/largeEnv-4                   10.16Ki ± 0%   10.16Ki ± 0%       ~ (p=0.545 n=6)
BenchSaveResultsSeed/N=1000-4                1.278Gi ± 0%   1.278Gi ± 0%       ~ (p=0.589 n=6)
BenchSaveResultsUpdateUnchanged/N=1000-4     32.20Mi ± 0%   32.22Mi ± 0%       ~ (p=0.818 n=6)
BenchSaveResultsUpdateChanged/N=1000-4       797.2Mi ± 1%   797.1Mi ± 1%       ~ (p=0.818 n=6)
geomean                                      990.3Ki        990.4Ki       +0.01%
¹ all samples are equal

                                         │ bench-base.txt │            bench-head.txt            │
                                         │   allocs/op    │  allocs/op    vs base                │
LocationFilter/smallEnv-4                     278.0 ±  0%    278.0 ±  0%       ~ (p=1.000 n=6) ¹
LocationFilter/largeEnv-4                     522.0 ±  0%    522.0 ±  0%       ~ (p=1.000 n=6) ¹
RunTemplateBool/smallEnv-4                    216.0 ±  0%    216.0 ±  0%       ~ (p=1.000 n=6) ¹
RunTemplateBool/largeEnv-4                    297.0 ±  0%    297.0 ±  0%       ~ (p=1.000 n=6) ¹
BenchSaveResultsSeed/N=1000-4                442.7k ±  0%   442.7k ±  0%       ~ (p=1.000 n=6)
BenchSaveResultsUpdateUnchanged/N=1000-4     396.9k ±  0%   396.9k ±  0%       ~ (p=0.485 n=6)
BenchSaveResultsUpdateChanged/N=1000-4       984.1k ± 13%   984.1k ± 13%       ~ (p=0.937 n=6)
geomean                                      7.703k         7.703k        +0.00%
¹ all samples are equal

                                         │ bench-base.txt │           bench-head.txt           │
                                         │      MB/s      │    MB/s     vs base                │
BenchSaveResultsSeed/N=1000-4                0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
BenchSaveResultsUpdateUnchanged/N=1000-4     0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
BenchSaveResultsUpdateChanged/N=1000-4       0.000 ± 0%     0.000 ± 0%       ~ (p=1.000 n=6) ¹
geomean                                                 ²               +0.00%               ²
¹ all samples are equal
² summaries must be >0 to compute geomean

[coderabbitai]

Important

Review skipped

Draft detected.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: 57fc454b-00aa-47a4-ac18-fa2d0d65bcf0

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Commit unit tests in branch feat/cloudtrail-s3-source

✨ Simplify code

• Create PR with simplified code
• Commit simplified code in branch feat/cloudtrail-s3-source

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[adityathebe]

We'll use clickhouse scraper