Skip to content

TODO.md

Latest commit

 

History

History
81 lines (65 loc) · 3.92 KB

TODO.md

File metadata and controls

81 lines (65 loc) · 3.92 KB
Raw

TODO

Language / semantics

  • Fold: init + reduce instead of ?? initial-state pattern. Done. fold by <fields> init { ... } reduce { ... } is the new syntax. Init is evaluated once at pipeline compile time to (a) seed every group without re-running VRL at runtime and (b) provide a concrete Kind to the reduce block, so .count + 1 type-checks without any ?? 0 noise.

Performance

  • SIMD-accelerated search "literal" stage. Done (memchr::memmem).

  • Chunked events on channels. Done (CHUNK_SIZE = 1024).

  • Per-chunk shard batching in fold. Done. Events are bucketed by target shard before locking; each shard's mutex is taken exactly once per chunk. fold by [] (single shard) drops from O(events-per-chunk) lock acquisitions to O(1).

  • Stage fusion for adjacent pure-CPU stages. Done. Adjacent search / filter / map / since / until stages run inside one fused worker pool. Channel hops collapse N → 1, VRL scratch allocates once per fused worker, and events are retained in place between ops so a drop in op 1 skips op 2..N. Measured 8.9× faster than Python on a chained workload (vs 3.5× for single-filter count-error).

  • count fold primitive / no-VRL fast path. The most common accumulator (.count = .count + 1 with an integer init) could bypass VRL entirely — the fold stage recognises the shape at compile time and runs a pure Rust u64 += 1 per event under the shard lock. Useful both as a perf win and as a stepping stone for fused accumulators.

Features

  • Sort / top-K stage. Done. sort by <path> [desc] [limit N], with VRL-compiled path extraction so .state.count works. Full sort today; bounded heap for top-K deferred until a workload justifies it.

  • count sugar. Done. count and count by <fields> lower to the canonical counter fold at parse time so downstream optimisations (init pre-eval, shard batching) apply automatically.

  • Simpler in-flight accounting (per-stage events_in == events_out). Today the compiler has to know which fused stages are pre- vs post-discover so it can hand them the right counter (monitored vs unmonitored). The boundary distinction is load-bearing for correctness but ugly — it leaks the discover-termination invariant into the compiler's stage-emit loop. Alternative: every stage already maintains events_in / events_out atomics via StageStats. The monitor could poll a different invariant — "for every stage, events_in == events_out" — which is equivalent to pipeline-wide quiescence and doesn't care about where discover sits. Needs a clean no-poll wake path (same notify-on-zero trick, but signalled whenever a stage's delta reaches zero) and a cheap check across the stats vector.

  • Filename-timestamp early drop. Log files commonly carry date/time tokens in their names (app.log.2024-01-15, access_log_20240115, 2024/01/15/access.log, app-2024-01-15T10-30-00.log). A pipeline with since / until can use that to prune entire files at the glob / read boundary — never open or mmap them at all. For a year of daily logs on a "last 24 h" query, this turns 365 file reads into 1. Either a heuristic (find a date-ish pattern in the path) or a user-supplied filename_ts "%Y-%m-%d" hint passed to glob / read_lines. Composes with the existing per-line since byte-scan: files that straddle the boundary still get line-level filtering.

  • Time-window filter. Done. since / until stages with four extractor forms: .field (JSON needle scan), @leading (unix-int or ISO prefix), @bracketed (first [...]), and @clf (Apache CLF). Relative ("1h ago") and absolute time values resolved once at pipeline compile time. Per-event cost ~5 ns on the text corpus. Syslog (Mon DD HH:MM:SS, no year) and custom strftime formats left for later.

Other

  • Stage::Discover currently todo!()s — planned Phase 4.