Skip to content

out_influxdb: documentation for tag prefix stripping#1468

Open
ueli-g wants to merge 10 commits into
fluent:masterfrom
ueli-g:influxdb-strip-tag-prefix
Open

out_influxdb: documentation for tag prefix stripping#1468
ueli-g wants to merge 10 commits into
fluent:masterfrom
ueli-g:influxdb-strip-tag-prefix

Conversation

@ueli-g
Copy link
Copy Markdown

@ueli-g ueli-g commented Oct 1, 2024

Documentation to go with fluent/fluent-bit#9427

This removes a defined prefix from measurement names which might otherwise be shared between many measurements in the same data bucket. Compatibility is not affected, default behavior (with documented default strip_prefix "") is to strip zero character from the front which identical to the existing mechanism.

When writing to a range of different buckets, routing to the corresponding out_influxdb instances happens on tag matches. This change allows to match on tag prefixes, but strip them from the measurement name. This avoids having identical prefixes for all measurement names in the same data bucket.

To achieve this, read from char tag[] with an offset when writing the measurement name, provided the prefix matches the tag completely and the overlap is at most tag_length - 1 characters.

Summary by CodeRabbit

  • New Features
    • Added a strip_prefix option for InfluxDB output to remove exact leading prefixes from tag-derived measurement names, producing normalized measurements (e.g., stripping "cpu." or "gpu.").
  • Documentation
    • Updated docs with a "Prefix stripping" section and examples showing configuration and resulting measurement/bucket behavior.

Ueli Graf added 2 commits October 1, 2024 09:42
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
@patrick-stephens
Copy link
Copy Markdown
Contributor

@fluent/chronosphere-tech-writers should we be ensuring we add YAML config as well now always?

@eschabell eschabell self-assigned this Oct 22, 2025
@eschabell eschabell requested a review from esmerel October 22, 2025 00:17
@eschabell eschabell added waiting-for-user Waiting for user/contributors feedback or requested changes conflict Waiting on conflict to be resolved by contributor waiting-on-review Waiting on a review from mainteners labels Oct 22, 2025
@eschabell
Copy link
Copy Markdown
Collaborator

@esmerel can you review please, but leave open as code PR fluent/fluent-bit#9427 must merge first and is still open.

@eschabell
Copy link
Copy Markdown
Collaborator

eschabell commented Oct 22, 2025

@ueli-g can you resolve the conflicts and provide YAML config example (if you need help with that, happy to provide)? I also nudged you on the code PR fluent/fluent-bit#9427

Signed-off-by: Lynette  Miles <6818907+esmerel@users.noreply.github.com>
@esmerel
Copy link
Copy Markdown
Contributor

esmerel commented Oct 22, 2025

I fixed the merge conflicts and made review updates while I was fixing it since I was already in there.

@esmerel esmerel added waiting-on-code-merge and removed conflict Waiting on conflict to be resolved by contributor waiting-on-code-merge waiting-for-user Waiting for user/contributors feedback or requested changes waiting-on-review Waiting on a review from mainteners labels Oct 22, 2025
Comment thread pipeline/outputs/influxdb.md Outdated
Comment thread pipeline/outputs/influxdb.md Outdated
Signed-off-by: Lynette  Miles <6818907+esmerel@users.noreply.github.com>
Copy link
Copy Markdown
Contributor

@patrick-stephens patrick-stephens left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Config example needs updating to latest standard.

Comment thread pipeline/outputs/influxdb.md Outdated
Comment thread pipeline/outputs/influxdb.md Outdated
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
@ueli-g
Copy link
Copy Markdown
Author

ueli-g commented Nov 9, 2025

@eschabell thank you for the review. I added a yaml config example as requested and added more information on the behaviour. please let me know if there is anything else I can do to move this forward.

@ueli-g
Copy link
Copy Markdown
Author

ueli-g commented Nov 9, 2025

Config example needs updating to latest standard.

@patrick-stephens can you please have another look and let me know if this is more appropriate?

@eschabell
Copy link
Copy Markdown
Collaborator

@ueli-g would you mind sorting out the conflicts now showing up for this PR?

@eschabell eschabell added the conflict Waiting on conflict to be resolved by contributor label Apr 19, 2026
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
@ueli-g ueli-g requested a review from eschabell as a code owner April 20, 2026 09:54
@coderabbitai
Copy link
Copy Markdown
Contributor

coderabbitai Bot commented Apr 20, 2026

Warning

Rate limit exceeded

@ueli-g has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 58 minutes and 3 seconds before requesting another review.

To continue reviewing without waiting, purchase usage credits in the billing tab.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 2a13b0d2-6cc0-4e65-94b6-c413091b91c5

📥 Commits

Reviewing files that changed from the base of the PR and between acf18a3 and eb39179.

📒 Files selected for processing (1)
  • pipeline/outputs/influxdb.md
📝 Walkthrough

Walkthrough

Added documentation for a new strip_prefix configuration parameter to the InfluxDB output plugin that removes an exact leading prefix from event tags when deriving InfluxDB measurement names; includes behavior notes and YAML and fluent-bit.conf examples.

Changes

InfluxDB Output Documentation

Layer / File(s) Summary
Data Shape / Config API
pipeline/outputs/influxdb.md
Introduces the strip_prefix configuration parameter and documents its default "" value and exact-prefix matching semantics in the config table.
Behavior Explanation
pipeline/outputs/influxdb.md
Adds a "Prefix stripping" section describing how stripped prefixes affect measurement name derivation and bucket contents.
Examples — YAML
pipeline/outputs/influxdb.md
Adds a YAML example showing multiple inputs with tags and outputs using strip_prefix (cpu., gpu.) to normalize measurement names to one and two per bucket.
Examples — fluent-bit.conf
pipeline/outputs/influxdb.md
Adds equivalent fluent-bit.conf example demonstrating Strip_prefix usage and resulting measurement naming behavior.
Notes / Outcome
pipeline/outputs/influxdb.md
Explains resulting buckets and measurement names with and without prefix stripping (two measurement streams per bucket vs. normalized names).

Estimated code review effort

🎯 2 (Simple) | ⏱️ ~10 minutes

Suggested labels

5.0.3

Suggested reviewers

  • patrick-stephens

Poem

🐰 I nibble tags with gentle paws,
I peel the prefixes just because,
From cpu.one to one so neat,
Measurement names now skip a beat —
~CodeRabbit

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.
Title check ✅ Passed The title directly and specifically describes the main change: adding documentation for the tag prefix stripping feature in the InfluxDB output plugin.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
🧪 Generate unit tests (beta)
  • Create PR with unit tests

Tip

💬 Introducing Slack Agent: The best way for teams to turn conversations into code.

Slack Agent is built on CodeRabbit's deep understanding of your code, so your team can collaborate across the entire SDLC without losing context.

  • Generate code and open pull requests
  • Plan features and break down work
  • Investigate incidents and troubleshoot customer tickets together
  • Automate recurring tasks and respond to alerts with triggers
  • Summarize progress and report instantly

Built for teams:

  • Shared memory across your entire org—no repeating context
  • Per-thread sandboxes to safely plan and execute work
  • Governance built-in—scoped access, auditability, and budget controls

One agent for your entire SDLC. Right inside Slack.

👉 Get started


Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

Copy link
Copy Markdown
Contributor

@coderabbitai coderabbitai Bot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🧹 Nitpick comments (1)
pipeline/outputs/influxdb.md (1)

154-164: Consider clarifying the input configuration in the example.

All four inputs use name: cpu even though the tags suggest different hardware types (gpu). While this is technically valid and simplifies the example to focus on routing and prefix stripping, it might be slightly confusing. Consider adding a brief comment explaining that the same input plugin is used with different tags for demonstration purposes, or acknowledging that in a real scenario, gpu tags might come from different sources.

💡 Optional clarification

You could add a comment above the inputs section like:

 ```yaml
 pipeline:
   inputs:
+    # Using cpu plugin with different tags to demonstrate routing and prefix stripping
     - name: cpu
       tag: cpu.one

Or in prose before the example:

-When collecting data from many inputs into many buckets, it can be helpful to remove a common prefix using `Strip_prefix`.
+When collecting data from many inputs into many buckets, it can be helpful to remove a common prefix using `Strip_prefix`. The following example uses the same input plugin with different tags to demonstrate how prefix stripping works with pattern-based routing.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@pipeline/outputs/influxdb.md` around lines 154 - 164, The example under
pipeline.inputs uses name: cpu for all four entries which can confuse readers
because tags include gpu; update the example to either (a) add a brief inline
comment above the inputs (referencing pipeline.inputs and the name: cpu entries)
explaining that the cpu plugin is intentionally reused with different tags for
demonstration of routing/prefix stripping, or (b) change the relevant entries to
use a more representative name (e.g., name: gpu) for gpu.* tags so tag names and
input plugin names align; ensure the comment or rename clearly references the
tag values (cpu.one, cpu.two, gpu.one, gpu.two) so readers understand the
intent.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Inline comments:
In `@pipeline/outputs/influxdb.md`:
- Line 25: The documentation shows an inconsistency for the option strip_prefix:
the description says it defaults to an empty string but the Default column shows
"_none_"; update the Default column entry for strip_prefix to an explicit
empty-string representation (e.g., `""`) so it matches the textual description
and makes the default behavior clear; locate the table row referencing
strip_prefix and change the Default cell to `""` while leaving the description
and other cells unchanged.
- Around line 186-213: The fluent-bit.conf influxdb OUTPUT examples are missing
the Host and Port parameters shown in the YAML example; update both influxdb
OUTPUT blocks (the ones matching "cpu*" and "gpu*") to include Host and Port
entries (e.g., Host and Port keys alongside Name, Match, Bucket, and
Strip_prefix) so the two examples are consistent and clearly show the connection
endpoint settings.

---

Nitpick comments:
In `@pipeline/outputs/influxdb.md`:
- Around line 154-164: The example under pipeline.inputs uses name: cpu for all
four entries which can confuse readers because tags include gpu; update the
example to either (a) add a brief inline comment above the inputs (referencing
pipeline.inputs and the name: cpu entries) explaining that the cpu plugin is
intentionally reused with different tags for demonstration of routing/prefix
stripping, or (b) change the relevant entries to use a more representative name
(e.g., name: gpu) for gpu.* tags so tag names and input plugin names align;
ensure the comment or rename clearly references the tag values (cpu.one,
cpu.two, gpu.one, gpu.two) so readers understand the intent.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: d0e3e2e3-0f34-4198-bac2-fddc13977b39

📥 Commits

Reviewing files that changed from the base of the PR and between 343903f and acf18a3.

📒 Files selected for processing (1)
  • pipeline/outputs/influxdb.md

Comment thread pipeline/outputs/influxdb.md Outdated
Comment thread pipeline/outputs/influxdb.md
Ueli Graf added 2 commits April 20, 2026 11:59
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
@ueli-g
Copy link
Copy Markdown
Author

ueli-g commented Apr 20, 2026

@edsiper thank you for your review, updated accordingly.

This has remained open for a while now, please let me know if there is anything that I can do or that I am missing to help move this and the corresponding feature in fluent-bit along.

@eschabell eschabell removed the conflict Waiting on conflict to be resolved by contributor label Apr 24, 2026
Comment thread pipeline/outputs/influxdb.md Outdated
Comment thread pipeline/outputs/influxdb.md Outdated
@eschabell eschabell added the waiting-on-review Waiting on a review from mainteners label Apr 24, 2026
Ueli Graf added 2 commits May 6, 2026 18:54
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
Signed-off-by: Ueli Graf <ueli@9tlabs.com>
@ueli-g
Copy link
Copy Markdown
Author

ueli-g commented May 6, 2026

@eschabell updated accordingly

@patrick-stephens patrick-stephens requested a review from eschabell May 8, 2026 09:22
Copy link
Copy Markdown
Collaborator

@eschabell eschabell left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ueli-g thanks for the docs PR, just waiting now on code PR to merge.

@eschabell eschabell removed the waiting-on-review Waiting on a review from mainteners label May 8, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants