-
Notifications
You must be signed in to change notification settings - Fork 84
Move startup helpers into library crates for external embedding #2462
Description
Pre-filing checklist
- I searched existing issues and didn't find a duplicate
Component(s)
Rust OTAP dataflow (rust/otap-dataflow/)
Is your feature request related to a problem?
Summary
The engine crates are already usable as libraries - Controller, OtelDataflowSpec, PipelineFactory, and node registration via linkme all work from external binaries. That's great.
The gap is that several practical startup functions live only in src/main.rs and can't be reused without copying them:
-
validate_engine_components()- checks that every node URN in the config maps to a registered component and runs per-component config validation. This is the kind of thing every
embedding binary needs, and it only uses public APIs (PipelineFactorygetters,NodeKind, factory entryvalidate_config). -
apply_cli_overrides()- applies core allocation and admin bind overrides to anOtelDataflowSpec. Uses only public types (CoreAllocation,HttpAdminSettings,ResourcesPolicy). -
system_info()- prints registered components and system info. Useful for diagnostics in any distribution.
These are small, stable functions with no private dependencies. Moving them into a library module (e.g. otap_df_controller::startup or a thin otap-df-cli crate) would let custom binaries share them without copying code that may drift across releases.
This would make the engine meaningfully easier to embed in custom distributions - the same pattern used by projects like bindplane-otel-collector in the Go ecosystem.
Proposed Solution
- Added above.
Alternatives Considered
-
Copy the functions into each downstream binary.: It works, but the functions are non-trivial (~80 lines combined), touch config internals, and would silently diverge from upstream if validation logic changes. Maintainable for one binary, less so for several.
-
Make
src/main.rsitself a thin wrapper over library calls. This is effectively the proposed change -main.rswould still exist and work the same way, it would just call into library functions instead of defining them inline. No behavioral change for existing users. -
Publish a separate
otap-df-cliutility crate. Cleaner separation, but probably overkill for three functions. A module insideotap-df-controllerorotap-df-configis simpler and avoids a new crate.
Additional Context
No response
Metadata
Metadata
Assignees
Labels
Type
Projects
Status