LLVM `readelf` is slower than GNU `readelf`

[robertkirkman]

Hi, LLVM readelf is slightly slower than GNU readelf on the same computers and inputs.

Average "real" time of 3 runs of time readelf -d /usr/bin/ls: 5.33 ms
Average "real" time of 3 runs of time llvm-readelf -d /usr/bin/ls: 10 ms

Computer specifications:

OS: Gentoo GNU/Linux
LLVM readelf version: 21.1.8
GNU readelf version: 2.45.1
CPU: AMD Ryzen 9 5950X
RAM: 80 GB DDR4-2133
SSD: Samsung 990 PRO NVMe

[TomJo2000]

Just wanna pop in here to add a little more context.

This issue was previously known but deemed as insignificant to us (Termux as a project) at the time.
It has now become a larger problem for one of our downstreams, PyInstaller

• Attempt to restore compatibility with Termux python 3.13 pyinstaller/pyinstaller#9398

after we overhauled the way our build system utilizes Binutils to standardize on the LLVM implementations.

• rmpkg(main/binutils-is-llvm): make LLVM binutils default in Termux termux/termux-packages#28586

[robertkirkman]

Clarification: this was not previously known.

Also, it is no longer an immediate problem for PyInstaller, because in the current situation, they will be using GNU readelf again.

[aengelke]

On my system, a statically linked -fPIE llvm-readelf is 1.8x faster than binutils -fPIE readelf, but a dynamically linked llvm-readelf is 2.5x slower.

Root causes appear to be:

• Large number of page faults (~75% system time).
  • 40% come from ld.so processing the ~170k relocations and doing symbol lookups.
  • 50% come from libLLVM.so global constructors, primarily for initializing cl::opt. (NB: both accessing the global constructor code and accessing the cl::opt cause page faults.) (NB: a cl::opt is in .bss and tends to be >600 bytes. All initialization is done in a global constructor.) (NB: this is especially bad for dylib builds, where all cl::opt are always initialized, not just the ones from the needed components.)
  • 6% come from llvm-readelf global constructors.
• User CPU time (~25%) is largely spend on relocations and symbol lookups (15%) and executing global constructors, again largely for cl::opt (8%).

For faster dylib startup times, we would need to re-architect cl::opt to move away from global constructors and reduce the number of dynamic relocations.

[robertkirkman]

I can confirm that statically linking LLVM readelf to libLLVM makes its performance very close to the performance of GNU readelf, and that applying the same setting (-DLLVM_LINK_LLVM_DYLIB=OFF) to the entire LLVM build also improves the performance of other LLVM tools, because it significantly improved the three-run average build time of the Mesa graphics driver with clang with all other settings on the same system being unchanged,

however, applying -DLLVM_LINK_LLVM_DYLIB=OFF globally vastly increases the size of many of the LLVM executables, and unfortunately, -DLLVM_TOOL_LLVM_DRIVER_BUILD=ON does not sufficiently mitigate the size increase when LLVM is being built with many features enabled, because many of the executables are unaffected by that setting.

I think one option might be for me to build two separate LLVM CMake sessions, one with the static linking to libLLVM and the symbolic linking to a single binary both enabled and the other with them both disabled, and then mix the resulting builds together taking everything that became symbolically linked to the llvm binary from one of the builds and taking everything else that wasn't from the other build that has smaller executables. However, that could create a very large and difficult-to-maintain mess in the build script code.