Skip to content

BASELINE.md

Latest commit

 

History

History
67 lines (49 loc) · 3.25 KB

BASELINE.md

File metadata and controls

67 lines (49 loc) · 3.25 KB
Raw

Baseline (pre-optimization)

Initial numbers captured after the first hyperfine run against all three workloads. Flow uses per-event VRL for every filter / map / fold stage, which is the target for the next round of optimization work (see ../../TODO.md).

count-error — substring filter over text logs

Workload: count lines containing the literal ERROR across text/.log files.

corpus flow python powershell flow speedup vs python
text/small (6 MB, 100k lines) 41.5 ms ± 15.2 167.1 ms ± 2.8 276.4 ms ± 2.3 4.03 ×
text/medium (56 MB, 1M lines) 249.6 ms ± 44.9 435.1 ms ± 10.4 737.2 ms ± 6.2 1.74 ×
text/large (168 MB, 3M lines) 786.0 ms ± 83.3 1.043 s ± 0.025 1.741 s ± 0.017 1.33 ×

Observation: flow's advantage over Python narrows as data scales up — the per-event VRL overhead (~2.7 µs/event for a trivial contains) dominates once fixed startup cost amortizes away. This is the workload the planned SIMD search stage targets.

count-all — discovery + read over many tiny JSON files

Workload: count records across json-small/**/*.json (one object per file).

corpus files flow python powershell flow speedup vs python
json-small/small 1 k 24.5 ms ± 1.4 186.5 ms ± 2.7 193.1 ms ± 2.3 7.60 ×
json-small/medium 10 k 82.8 ms ± 7.7 598.7 ms ± 17.6 362.1 ms ± 3.4 7.23 ×
json-small/large 100 k 751.4 ms ± 58.7 5.201 s ± 0.056 2.363 s ± 0.027 6.92 ×

Observation: flow's parallel walker + mmap reading holds a ~7× lead over Python across all sizes. No VRL on the hot path means the per-event overhead doesn't dominate. This is where the current architecture shines.

group-by-level — JSON parse + fold

Workload: parse_json! each line and aggregate counts by .level.

corpus lines flow python powershell flow speedup vs python
jsonl/small (5.5 MB) 40 k 46.9 ms ± 6.3 246.4 ms ± 2.7 3.022 s ± 0.021 5.26 ×
jsonl/medium (56 MB) 400 k 318.5 ms ± 16.5 1.220 s ± 0.021 27.596 s ± 0.082 3.83 ×

Observation: PowerShell's ConvertFrom-Json per-line cost is ~order-of- magnitude worse than Python's stdlib json.loads. Flow beats Python ~4–5 × but not by the wide margin json-small shows — per-event VRL cost for parse_json! + fold is substantial and will narrow further on larger corpora unless addressed.

Per-stage instrumentation snapshot

FLOW_TIMING=1 ./target/release/flow --script-file bench.dsl on text/medium:

stage in out work recv-wait send-wait
glob 0 101 0.00 ms 0.00 ms 0.01 ms
read_lines 101 1 M 10.62 ms 0.00 ms 4 497.23 ms
filter 1 M 200 k 2 687.09 ms 1 070.44 ms 853.28 ms
fold 200 k 1 2 365.36 ms 2 500.17 ms 0.01 ms

(All times aggregated across worker threads.)

Filter's work time / events = 2.7 µs/event. A raw memchr::memmem is ~tens of ns. ~100 × of that budget is spent building Value::Object and running the VRL runtime.