erts: add chunked process_info backtrace BIFs

[lmth]

Summary

Add three new BIFs for reading a process backtrace incrementally in
fixed-size chunks, without materialising the full backtrace in memory
at once:

• erlang:process_info_backtrace_start(Pid, ChunkSize) → Handle
• erlang:process_info_backtrace_next(Handle) → {{backtrace, Chunk}} | done
• erlang:process_info_backtrace_stop(Handle) → ok

Motivation

We are stuck trying to make it possible to dump out the backtraces of
certain erlang processes in production systems, and for those to be complete.

When our customers deploy our products, and they run into issues (often our
bugs or misarchitectures, in erlang code) where their erlang node grows
non-catastrophically, but nearing an OOM situation, we provide a built-in
"debug dump" functionality, which writes diagnostics on file.
Those diagnostics include information about allocators and process info etc.
This is similar to a crash dump but intended to be non-destructive.
Unfortunately, we have run into situations where our debug dump has tipped
the scales, due to the amount of memory required to run
process_info(Pid,backtrace) -> BigBinary
with a big process as target.

For the time being, we choose to skip processes over a certain stack depth
and/or heap size.

But, those processes are usually the most interesting processes given the
problem statement.

So, to make it possible for us to capture the backtraces of really big
processes, and without any truncation (unless decided by our debug dump
collector on a timeout basis) we would like to suggest a chunked API
for reading process backtraces.

With this feature, plus the new process_iterator APIs, users would be
able to reduce the memory footprint of their system diagnostics tools built
on top of process_info.

This is a fair amount of code in erts, and it is more code than the PR
author has managed to get the full grasp of, so look upon this PR more
as a request for functionality than a suggestion for inclusion in the
OTP code base as-is.

Implementation notes

• A resumable ErtsPrintTermCursor term printer (erl_printf_term.c/h)
  yields after printing at most ChunkSize bytes of any single term, then
  resumes from the same position on the next call.
• The session handle is a magic binary so the target process is automatically
  unblocked if the caller drops the handle without calling _stop.
• The stepbuf allocator was changed from ERTS_ALC_T_TMP_DSBUF (scheduler-
  local) to ERTS_ALC_T_SIG_DATA (scheduler-safe) to handle the case where
  _next/_stop runs on a different scheduler than _start.

Tests

15 test cases in process_SUITE, group chunked_backtrace_bif.

Checklist

• Commit subjects ≤ 72 chars, no trailing period
• SPDX license headers on new files
• git diff --check whitespace clean
• otp_build check --no-dialyzer --no-docs --no-tests passes
• -doc since tags: ~"OTP @OTP-XXXXX@" — replaced automatically by OTP release tooling
• Full CI (dialyzer + docs) via GitHub Actions

[github-actions]

CT Test Results

    4 files    199 suites   1h 53m 58s ⏱️
3 375 tests 2 902 ✅ 472 💤 1 ❌
4 275 runs  3 725 ✅ 549 💤 1 ❌

For more details on these failures, see this check.

Results for commit 5de641b.

♻️ This comment has been updated with latest results.

To speed up review, make sure that you have read Contributing to Erlang/OTP and that all checks pass.

See the TESTING and DEVELOPMENT HowTo guides for details about how to run test locally.

Artifacts

• Complete CT logs (Download Logs)
• HTML Documentation (Download HTML Docs)
• No Windows Installer found

// Erlang/OTP Github Action Bot

[jhogberg]

Thanks for the PR, we'll look into it. 🙂

We are stuck trying to make it possible to dump out the backtraces of
certain erlang processes in production systems, and for those to be complete.

A backtrace is mostly just expensive when the stack is really deep, and in that case you want it to be complete because the most interesting things usually are in the beginning ("how did we enter this catastrophic stack growth?"). Do you think a "front-trace" giving the first N frames would be sufficient in most cases? It would be a much smaller change.

[lmth]

Thanks for the PR, we'll look into it. 🙂

That's highly appreciated. 🙏

We are stuck trying to make it possible to dump out the backtraces of
certain erlang processes in production systems, and for those to be complete.

A backtrace is mostly just expensive when the stack is really deep, and in that case you want it to be complete because the most interesting things usually are in the beginning ("how did we enter this catastrophic stack growth?"). Do you think a "front-trace" giving the first N frames would be sufficient in most cases? It would be a much smaller change.

We take what we get, of course. In the cases where we now have to skip processes from the dump, we would use that functionality instead.

Maybe a reentrant term printer could be useful elsewhere? If you implemented that in a more trustworthy/robust way than what is in this PR, then "the rest" is not so much. 😉

We have suffered on numerous occasions, that the important data is hidden "later" in very deep terms in the
backtraces, and then we have changed the records of our gen_... processes to put the "not expected to be very
interesting, but potentially taking a lot of text estate when printed" terms later in the process' state record.

This is a different situation from "if we print it we die", but it leads to the same problem - we don't always get to
see the interesting data, when we didn't guess (not a good word for describing how experience leads to better
design) a good layout in advance. So, the suggested capability is not just taken out of thin air, even though the
function names are too long. Sorry about those.

[jhogberg]

That's highly appreciated. 🙏

We've taken a glance, and it seems like we won't be able to take a deeper look until after summer, so it'll have to wait.