Customize core isolation for critical RT applications

This guide focuses on the kontron-albl-imx8mm system but can be applied to grisp2 and maybe other future platforms.

Customize core isolation

When building the firware it is possible to setup u-boot to exclude a subset of the CPU cores so that the kernel will ignore them.

These are extra kernel parameters that you can customize here: https://github.com/grisp/grisp_alloy/blob/main/system_kontron-albl-imx8mm/fwup.conf

Check this example https://github.com/grisp/grisp_alloy/blob/ab17aac50c58bbeebda86d3e17c66c70e5dbff30/system_kontron-albl-imx8mm/fwup.conf#L269

Explanation of real-time parameters

uboot_setenv(uboot-env, "bootargs_extra", "isolcpus=3 nohz_full=3 rcu_nocbs=3 rcu_nocb_poll intel_idle.max_cstate=1 processor.max_cstate=1")

The bootargs_extra U-Boot environment variable sets kernel boot parameters:

• isolcpus=3 — isolates CPU 3 from the scheduler
• nohz_full=3 — disables tick on CPU 3
• rcu_nocbs=3 — excludes CPU 3 from RCU callback processing This reserves CPU 3 for real-time tasks, leaving CPUs 0-2 for normal system

rcu_nocb_poll

When CPUs are excluded from RCU callbacks (rcu_nocbs), callbacks are handled by dedicated kthreads. rcu_nocb_poll makes these kthreads poll for work instead of waiting for wake-ups. Reduces wake-up latency and improves real-time behavior.

intel_idle.max_cstate=1 and processor.max_cstate=1

These limit CPU idle states (C-states). max_cstate=1 restricts CPUs to C0 (active) and C1 (halt), preventing deeper sleep states (C2, C3, etc.). intel_idle.max_cstate=1 applies to Intel CPUs (likely ignored on ARM). processor.max_cstate=1 is a generic ACPI parameter that may apply on ARM systems. Why: Deeper C-states add wake-up latency, which hurts real-time determinism.

Example: change isolation to 2 cores

If you need to increase the number of isolated core you can just change the numbers here.

# Isolating cores 2 and 3, kernel will use 0 and 1
        uboot_setenv(uboot-env, "bootargs_extra", "isolcpus=2,3 nohz_full=2,3 rcu_nocbs=2,3 rcu_nocb_poll intel_idle.max_cstate=1 processor.max_cstate=1")

This will let core 2 and 3 available and free from kernel interference.

Deploy an Erlang or Elixir release on a specific core subset

To run an Erlang VM on a specific core you need to deploy multiple projects. As described in the README, only the first project will boot. This project will run on the cores that are left available to the kernel. In the case of this guide these will be cores 0 and 1.

All other projects are just deployed and will not be started by the firmware.

Learn how to run multiple VMs

To be able also run other Erlang VMs, please read this guide: https://github.com/grisp/grisp_alloy/wiki/Run-Multiple-VMs-with-grisp_supervisor

Select specific cores in grisp_supervisor

grisp_supervisor allows to specify various settings for each child to spawn.

By default it searches /etc/grisp_supervisor.config for configuration.

Tutorial

We assume you already built the toolchain and SDK.

If you changed the fwup.conf to change the core isolation parameters, you need to rebuild the SDK.

./build-sdk.sh -KPc kontron-albl-imx8mm

Clone grisp_supervisor where you like:

git clone https://github.com/grisp/grisp_supervisor.git

Build the 2 sample projects that ship with grisp_alloy, plus grisp_supervisor

./build-project.sh -Kc kontron-albl-imx8mm samples/hello_grisp
./build-project.sh -Kc kontron-albl-imx8mm samples/hello_elixir
./build-project.sh -Kc kontron-albl-imx8mm path/to/grisp_supervisor

Now we need do choose a directory that will be installed as filesystem overlay on the firmware image. This allows us to export files into the final image. I will use samples/overlay/ inside grisp_alloy for convenience.

Create the file samples/overlay/etc/grisp_supervisor.config.

grisp_supervisor will consult /etc/grisp_supervisor.config at runtime.

% /etc/grisp_supervisor.config
[
    {nodes, [
        {hello_grisp, #{
            root_dir => "/srv/alloy/hello_grisp",
            boot_script => "start"
        }},
        {hello_elixir, #{
            root_dir => "/srv/alloy/hello_elixir",
            boot_script => "start",
            cpu => "2,3", %% CPU 2 and 3 are isolated for RT
        }}
    ]}
].

Note: grisp_supervisor uses taskset.

The string you put into cpu is directly passed as --cpu-list argument and follows the taskset syntax.

Now that the projects are built and the overlay is ready, build the firmware.

./build-firmware.sh -Kc -n test -o samples/overlay kontron-albl-imx8mm grisp_supervisor hello_elixir hello_grisp

On boot:

1. grisp_supervisor starts
2. consults /etc/grisp_supervisor.config
3. all children are started one by one
   • hello_grisp will use the non-isolated cores (0,1)
   • hello_elixir will run exclusively on cores (2,3)