Skip to content

Commit 23149a2

Browse files
dergoeggeclaude
dergoegge
and
claude
committed
ci: add A/B benchmarking pipeline for fuzzamoto-libafl
Add a runs-on.com workflow that compares two fuzzer versions (e.g. a PR vs its base) on a statistically sound basis: run each version N times for a fixed duration in parallel, then aggregate and compare the campaigns. Bitcoin Core (the target) is built once into a shared image and held constant; only the bind-mounted fuzzamoto source differs between the two arms, so the comparison isolates the fuzzer change. - ci/libafl.justfile: split nyx_init out of compile_nyx and add a time-bounded `bench` recipe (built with the `bench` feature) that emits bench-cpu_000.csv snapshots. - ci/benchmark-evaluation.py: port of the smite evaluation script - keeps all stats (Mann-Whitney U, Holm-Bonferroni, Vargha-Delaney A12, AUC, final coverage, median execs/s, IQR + plots); reads fuzzamoto bench CSVs instead of AFL++ output. Union coverage is omitted (needs a raw map dump). - .github/workflows/benchmark.yml: setup/build/campaign(matrix version x run)/compare jobs. Triggered by workflow_dispatch (two arbitrary refs) or PR labels: 'needs benchmark' (full, N=10, 3600s) and 'needs benchmark smoke' (cheap wiring test, N=2, 300s). Runner specs are inlined in each job's runs-on label; campaign runners are pinned to one exact instance type (nested-virt for /dev/kvm, spot disabled) so all campaigns run on identical hardware. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
1 parent ea874f1 commit 23149a2

3 files changed

Lines changed: 872 additions & 1 deletion

File tree

‎.github/workflows/benchmark.yml‎

Lines changed: 210 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,210 @@
1+
name: Benchmark
2+
3+
# A/B benchmark of two fuzzamoto-libafl versions: run each version N times for a fixed
4+
# duration in parallel, then aggregate and statistically compare the campaigns.
5+
#
6+
# Bitcoin Core (the target) is built once into a shared image and held constant; only the
7+
# bind-mounted fuzzamoto source differs between the two arms, so the comparison isolates
8+
# the fuzzer change. See ci/benchmark-evaluation.py for the metrics.
9+
#
10+
# Runner specs are inlined in each job's `runs-on:` label (runs-on.com syntax); there is
11+
# no .github/runs-on.yml. Campaign runners are pinned to a single exact instance type so
12+
# every campaign runs on identical hardware (see the campaign job).
13+
#
14+
# Triggers:
15+
# - workflow_dispatch: compare two arbitrary refs (full control over N / duration).
16+
# - PR label 'needs benchmark': full run, N=10, 3600s/campaign (PR base vs head).
17+
# - PR label 'needs benchmark smoke': cheap wiring test, N=2, 300s/campaign.
18+
19+
on:
20+
workflow_dispatch:
21+
inputs:
22+
baseline_ref:
23+
description: "Baseline fuzzamoto git ref"
24+
default: "master"
25+
treatment_ref:
26+
description: "Treatment fuzzamoto git ref"
27+
required: true
28+
runs:
29+
description: "Campaigns per version (N)"
30+
default: "10"
31+
duration_secs:
32+
description: "Seconds per campaign"
33+
default: "3600"
34+
pull_request:
35+
# Gated on the 'needs benchmark' / 'needs benchmark smoke' labels (checked in setup's `if:`).
36+
types: [labeled]
37+
38+
permissions:
39+
contents: read
40+
pull-requests: write # for the PR comment
41+
42+
jobs:
43+
setup:
44+
if: github.event_name == 'workflow_dispatch' || github.event.label.name == 'needs benchmark' || github.event.label.name == 'needs benchmark smoke'
45+
runs-on:
46+
- runs-on=${{ github.run_id }}
47+
- cpu=2+4
48+
- family=m7i+c7i
49+
- image=ubuntu22-full-x64
50+
- extras=s3-cache
51+
outputs:
52+
baseline: ${{ steps.refs.outputs.baseline }}
53+
treatment: ${{ steps.refs.outputs.treatment }}
54+
duration: ${{ steps.refs.outputs.duration }}
55+
runs_json: ${{ steps.refs.outputs.runs_json }} # e.g. "[1,2,...,10]"
56+
steps:
57+
- id: refs
58+
run: |
59+
if [ "${{ github.event_name }}" = "pull_request" ]; then
60+
echo "baseline=${{ github.event.pull_request.base.sha }}" >> "$GITHUB_OUTPUT"
61+
echo "treatment=${{ github.event.pull_request.head.sha }}" >> "$GITHUB_OUTPUT"
62+
if [ "${{ github.event.label.name }}" = "needs benchmark smoke" ]; then
63+
N=2
64+
echo "duration=300" >> "$GITHUB_OUTPUT"
65+
else
66+
N=10
67+
echo "duration=3600" >> "$GITHUB_OUTPUT"
68+
fi
69+
else
70+
echo "baseline=${{ inputs.baseline_ref }}" >> "$GITHUB_OUTPUT"
71+
echo "treatment=${{ inputs.treatment_ref }}" >> "$GITHUB_OUTPUT"
72+
N=${{ inputs.runs }}
73+
echo "duration=${{ inputs.duration_secs }}" >> "$GITHUB_OUTPUT"
74+
fi
75+
echo "runs_json=$(seq -s, 1 "$N" | sed 's/^/[/;s/$/]/')" >> "$GITHUB_OUTPUT"
76+
77+
build:
78+
needs: setup
79+
# Fat instance to build the shared image (Bitcoin Core + AFL++ + depends).
80+
runs-on:
81+
- runs-on=${{ github.run_id }}
82+
- cpu=16+32
83+
- family=c7i+m7i+c7a
84+
- image=ubuntu22-full-x64
85+
- volume=120gb:gp3
86+
- extras=s3-cache # S3-backed docker layer cache (no 10GB cap)
87+
- spot=false
88+
steps:
89+
- uses: actions/checkout@v4 # Bitcoin Core build is version-independent
90+
- uses: docker/setup-buildx-action@v3
91+
- name: Build & cache the shared image
92+
uses: docker/build-push-action@v5
93+
with:
94+
context: .
95+
file: ./Dockerfile.libafl
96+
push: false
97+
tags: fuzzamoto-libafl:bench
98+
platforms: linux/amd64
99+
cache-from: type=gha
100+
cache-to: type=gha,mode=max # backed by runs-on s3-cache (no 10GB cap)
101+
102+
campaign:
103+
needs: [setup, build]
104+
# nested-virt exposes /dev/kvm (required by Nyx). IMPORTANT: pinned to a single,
105+
# exact instance type so every campaign (both arms, all N runs) runs on identical
106+
# hardware - otherwise a coverage/exec-rate delta could reflect instance variance
107+
# rather than the fuzzer change. spot=false so an interruption can't void a 1h run.
108+
runs-on:
109+
- runs-on=${{ github.run_id }}
110+
- family=c7i.2xlarge # exact type; adjust to one nested-virt type in your account
111+
- image=ubuntu22-full-x64
112+
- nested-virt
113+
- volume=80gb:gp3
114+
- extras=s3-cache
115+
- spot=false
116+
timeout-minutes: 90 # 60m campaign + build/compile headroom
117+
strategy:
118+
fail-fast: false # a dead campaign must not cancel the others
119+
matrix:
120+
version: [baseline, treatment]
121+
run: ${{ fromJSON(needs.setup.outputs.runs_json) }}
122+
steps:
123+
- name: Checkout fuzzer version under test
124+
uses: actions/checkout@v4
125+
with:
126+
ref: ${{ matrix.version == 'baseline' && needs.setup.outputs.baseline || needs.setup.outputs.treatment }}
127+
- uses: docker/setup-buildx-action@v3
128+
- name: Restore image from cache
129+
uses: docker/build-push-action@v5
130+
with:
131+
context: .
132+
file: ./Dockerfile.libafl
133+
load: true
134+
tags: fuzzamoto-libafl:bench
135+
cache-from: type=gha # near-instant; layers built by `build`
136+
- name: Run campaign
137+
run: |
138+
rm -rf /tmp/out && mkdir -p /tmp/out
139+
docker run --privileged --device=/dev/kvm \
140+
-e BENCH_DURATION=${{ needs.setup.outputs.duration }} \
141+
-v ./:/fuzzamoto -v /tmp/out:/tmp/out \
142+
fuzzamoto-libafl:bench just -f /ci/libafl.justfile bench
143+
- name: Collect results
144+
run: |
145+
mkdir -p artifact
146+
cp /tmp/out/bench/bench-cpu_000.csv artifact/ \
147+
|| echo "no bench csv (campaign may have died early)"
148+
cp -r /tmp/out/cpu_000/crashes artifact/crashes 2>/dev/null || true
149+
- uses: actions/upload-artifact@v4
150+
with:
151+
name: bench-${{ matrix.version }}-${{ matrix.run }}
152+
path: artifact
153+
retention-days: 14
154+
155+
compare:
156+
needs: [setup, campaign]
157+
if: always()
158+
# Lightweight runner for report aggregation; does not affect measurements.
159+
runs-on:
160+
- runs-on=${{ github.run_id }}
161+
- cpu=2+4
162+
- family=m7i+c7i
163+
- image=ubuntu22-full-x64
164+
- extras=s3-cache
165+
steps:
166+
- uses: actions/checkout@v4
167+
- uses: actions/download-artifact@v4
168+
with:
169+
pattern: bench-*
170+
path: dl
171+
- name: Arrange into baseline/treatment layout
172+
run: |
173+
shopt -s nullglob
174+
for d in dl/bench-baseline-*; do
175+
n=${d##*-}; mkdir -p results/baseline/ir/$n
176+
cp "$d/bench-cpu_000.csv" results/baseline/ir/$n/ 2>/dev/null || true
177+
done
178+
for d in dl/bench-treatment-*; do
179+
n=${d##*-}; mkdir -p results/treatment/ir/$n
180+
cp "$d/bench-cpu_000.csv" results/treatment/ir/$n/ 2>/dev/null || true
181+
done
182+
- name: Install evaluation dependencies
183+
run: pip install numpy pandas matplotlib seaborn scipy statsmodels tabulate
184+
- name: Evaluate
185+
run: |
186+
python3 ci/benchmark-evaluation.py results --out report --hours min
187+
{
188+
echo "## fuzzamoto-libafl benchmark"
189+
echo
190+
echo "Baseline: \`${{ needs.setup.outputs.baseline }}\` • Treatment: \`${{ needs.setup.outputs.treatment }}\`"
191+
echo
192+
cat report/evaluation_report.md
193+
} >> "$GITHUB_STEP_SUMMARY"
194+
- uses: actions/upload-artifact@v4
195+
with:
196+
name: benchmark-report
197+
path: report
198+
retention-days: 90
199+
- name: Comment on PR
200+
if: github.event_name == 'pull_request'
201+
uses: actions/github-script@v7
202+
with:
203+
script: |
204+
const fs = require('fs');
205+
const body = fs.readFileSync('report/evaluation_report.md', 'utf8');
206+
await github.rest.issues.createComment({
207+
...context.repo,
208+
issue_number: context.issue.number,
209+
body,
210+
});

0 commit comments

Comments
 (0)