docs: add FAQ.md (measurement precision, JIT deopt, cross-tool differences)#609
Merged
Merged
Conversation
…-tool result differences Adds a FAQ.md at the repo root addressing the three topics called out in issue tinylibs#190 (measurement precision, JS JIT de-optimization, and why results differ from other benchmarking tools), grounded in the README's existing timestampProvider/time documentation. Closes tinylibs#190
jerome-benoit
requested changes
Jul 22, 2026
jerome-benoit
left a comment
jerome-benoit
left a comment
Collaborator
There was a problem hiding this comment.
Thanks for taking this on — the structure maps cleanly to the three checklist items in #190 and the writing is concise.
Two claims contradict tinybench's actual behavior (warmup handling; GC listed as a deopt cause) and should be corrected before this lands, plus a few precision fixes noted inline.
Not blocking, repo-level: FAQ.md isn't linked from anywhere, so it's effectively undiscoverable. Please add ### [FAQ](./FAQ.md) under the README Docs section.
commit: |
- Warmup: Tinybench warms up by default in a separate phase excluded from the reported statistics (cross-tool bullet + variance guidance corrected). - Deopt: falls back to a lower execution tier (not necessarily the interpreter); GC pauses are a separate latency source, not a deopt cause; soften DevTools. - Use 'relative margin of error' for rme; soften the amplification claim. - Link Timestamp Providers section to reduce duplication; add FAQ to README Docs.
- Measurement-precision intro: a fully timer-saturated distribution does
not reliably inflate rme, so drop the "high margin of error" framing and
point to the authoritative signals instead — the 'warning' event and
Task.detectedResolution (README Timer Diagnostics section).
- auto provider: align wording with the README ("most precise available").
- Decouple diagnostics: detectedResolution is exposed on every run, while the 'warning' event is dispatched only on timer saturation (task.ts). - Precision mitigation: prefer the 'warning' event / detectedResolution as the reliable saturation signal; note rme is only a soft hint (it can be either very high or very low under saturation, utils.ts rme=moe/|mean|). - Name the now option as a plain-millisecond alternative to timestampProvider (mutually exclusive, converted internally). - JIT deopt: reframe DevTools as added overhead / disabled optimizations rather than a deopt cause.
Reflect that detectedResolution is undefined when no positive timer sample was measured (e.g. every sample overridden), matching the README Timer Diagnostics wording (task.ts sets it per valid run, undefined otherwise).
Remove the timestampProvider code example that duplicated the README Timestamp Providers section near-verbatim (including the trailing comment); link to that section for the example instead, to avoid maintenance drift.
jerome-benoit
approved these changes
Jul 22, 2026
jerome-benoit
left a comment
jerome-benoit
left a comment
Collaborator
There was a problem hiding this comment.
Approving. Docs-only PR (new FAQ.md + a README Docs link), reviewed line-by-line against the current default branch (main @ 0647d27, including the #571 timer-overhead/saturation/resolution feature):
timestampProvidershorthands (hrtimeNow/performanceNow/bunNanoseconds/auto), theautoper-runtime selection, and thenowoption (plain-ms function, converted internally, mutually exclusive withtimestampProvider) all match the code (src/utils.ts,src/bench.ts,src/types.ts).detectedResolution(exposed on each valid run,undefinedotherwise) and the'warning'saturation event are described as implemented (src/task.ts);rmeis correctly the relative margin of error, and the FAQ correctly notes it is only a soft hint (a saturated task can report a very high or very lowrme).- Warmup on-by-default and excluded from the reported statistics; the cited
Statisticsfields all exist. - Both README anchors (
#timer-diagnostics,#timestamp-providers) resolve.
All review threads have been addressed and resolved. No code changes.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Closes #190 — adds a
FAQ.mdat the repo root covering the three topics in the issue's checklist:timestampProvidershorthands (hrtimeNow/performanceNow/bunNanoseconds/auto), increasetime, and watch therme.The content is grounded in the README's existing documentation of
timestampProviderand thetimeoption, and the issue author (@jerome-benoit) explicitly invited a PR for this in #143.AI usage
An AI assistant was used to help locate the relevant code and draft this change. I have reviewed and understand every line of the diff, and I am the sole author of this contribution.
Closes #190