diff --git a/ansible/roles/system/tasks/main.yml b/ansible/roles/system/tasks/main.yml index 045dcdd8b..bbae7c959 100644 --- a/ansible/roles/system/tasks/main.yml +++ b/ansible/roles/system/tasks/main.yml @@ -1,6 +1,16 @@ --- - name: Configure boot partition (/boot/{config,cmdline}.txt) - ansible.builtin.include_tasks: boot.yml + ansible.builtin.include_tasks: + file: boot.yml + # Without apply:, Ansible's --tags filter would let the include + # itself match (so the tasks get *included*) but then filter every + # task inside boot.yml back out — touches-boot-partition would + # become a no-op. apply: copies these tags onto each included + # task so they actually run when the include matches. + apply: + tags: + - touches-boot-partition + - raspberry-pi tags: - touches-boot-partition - raspberry-pi diff --git a/ansible/roles/system/templates/cmdline.txt.j2 b/ansible/roles/system/templates/cmdline.txt.j2 index 0a146a9fb..64895006a 100644 --- a/ansible/roles/system/templates/cmdline.txt.j2 +++ b/ansible/roles/system/templates/cmdline.txt.j2 @@ -3,6 +3,15 @@ here as a token list so additions/removals stay easy to review. Layout: Anthias-managed tokens first, then any tokens preserved from cmdline.txt.orig (e.g. cfg80211.* from raspi-config). + + Pi 5 4K HEVC CMA: NOT done via cmdline. Adding ``cma=512M`` here + makes the kernel take the cmdline value over the device-tree + ``linux,cma`` node, orphaning rpi-hevc-dec entirely (returns + ``Failed to probe hardware -517`` and ``/dev/video*`` disappears, + killing HEVC HW at every resolution). The CMA bump lives in + config.txt.j2 instead — ``dtoverlay=cma,cma-512`` rewrites the + DT linux,cma size in place, which preserves the codec driver's + ``memory-region`` reference. -#} {%- set tokens = [ 'console=serial0,115200', diff --git a/ansible/roles/system/templates/config.txt.j2 b/ansible/roles/system/templates/config.txt.j2 index 5bc0681ca..90bb11e09 100644 --- a/ansible/roles/system/templates/config.txt.j2 +++ b/ansible/roles/system/templates/config.txt.j2 @@ -16,7 +16,19 @@ dtparam=audio=on camera_auto_detect=1 display_auto_detect=1 auto_initramfs=1 +{# vc4-kms-v3d carries the CMA-size parameter on Pi 4/5; we override +the default on Pi 5 to give Hantro G2 enough buffer pool headroom for +4K HEVC dst frames. The standalone `dtoverlay=cma,cma-512` route +silently no-ops on Pi 5 because vc4-kms-v3d initialises the CMA +region first; reusing the v3d overlay's own cma-512 param is the +documented merge. Pi 4 already gets 512 MB CMA by default so the +override is a no-op there; the Pi 5 conditional below keeps the +extra arg local to the board that strictly needs it. #} +{% if device_type == 'pi5' -%} +dtoverlay=vc4-kms-v3d,cma-512 +{%- else -%} dtoverlay=vc4-kms-v3d +{%- endif %} max_framebuffers=2 # Don't have the firmware add a video= line to cmdline.txt — let the diff --git a/bin/generate_board_enablement_testbed.sh b/bin/generate_board_enablement_testbed.sh new file mode 100755 index 000000000..66e8af461 --- /dev/null +++ b/bin/generate_board_enablement_testbed.sh @@ -0,0 +1,147 @@ +#!/usr/bin/env bash +# Generate the 8-clip board-enablement test pack used by +# docs/board-enablement.md. +# +# Output: 4 H.264 + 4 HEVC clips at 1080p30/60 + 4K30/60, each ~1 +# minute. That's long enough to read a stable Dropped: count and +# capture mpv's hwdec-current banner, short enough that re-encoding +# the whole pack on a Pi 4 doesn't take an afternoon. +# +# Idempotent: clips that already exist (and pass an ffprobe sanity +# check) are skipped. Re-run after a power cycle and the script +# only redoes what's missing. +# +# Usage: +# bash bin/generate_board_enablement_testbed.sh [DEST_DIR] +# +# DEST_DIR defaults to ~/bbb-testbed. Set EARLY_EXIT=1 to stop on +# the first missing output (useful when iterating on a single clip +# in CI). + +set -euo pipefail + +DEST="${1:-$HOME/bbb-testbed}" +CUT_SECONDS="${CUT_SECONDS:-60}" +HEVC_CRF="${HEVC_CRF:-23}" +BBB_BASE='https://download.blender.org/demo/movies/BBB' + +mkdir -p "$DEST" +cd "$DEST" + +log() { printf '[testbed] %s\n' "$*" >&2; } + +# Returns 0 if $1 exists and ffprobe can parse it. Used so a +# previous half-written file (power cycle, ctrl-C) doesn't get +# silently kept on a re-run. +file_ok() { + [[ -s "$1" ]] || return 1 + ffprobe -v error -show_format -of default=nw=1:nk=1 "$1" \ + > /dev/null 2>&1 || return 1 + return 0 +} + +# Sources: H.264 + AAC, full-length, from blender.org. We trim +# rather than downloading shorter variants because the trims also +# exercise the upload path's handling of files whose container +# trailer isn't strictly mp4-spec-aligned (ffmpeg's mp4 muxer is +# tolerant; the Pi V3D V4L2 M2M decoder less so). +SOURCES=( + bbb_sunflower_1080p_30fps_normal.mp4 + bbb_sunflower_1080p_60fps_normal.mp4 + bbb_sunflower_2160p_30fps_normal.mp4 + bbb_sunflower_2160p_60fps_normal.mp4 +) + +log "step 1: download originals (skips existing)" +for f in "${SOURCES[@]}"; do + if file_ok "$f"; then + log " $f: present" + continue + fi + url="$BBB_BASE/$f" + log " $f: downloading from $url" + curl -sSL --fail -o "$f.tmp" "$url" + mv "$f.tmp" "$f" +done + +# Step 2: cut H.264 sources to CUT_SECONDS via -c copy. Avoids a +# re-encode (instant on any host), keeps the bitstream identical +# so the resulting clip exercises the same V3D / Hantro G1 path as +# the full-length original. +declare -A H264_TARGETS=( + [bbb_1080p_30fps.mp4]=bbb_sunflower_1080p_30fps_normal.mp4 + [bbb_1080p_60fps.mp4]=bbb_sunflower_1080p_60fps_normal.mp4 + [bbb_4k_30fps.mp4]=bbb_sunflower_2160p_30fps_normal.mp4 + [bbb_4k_60fps.mp4]=bbb_sunflower_2160p_60fps_normal.mp4 +) + +log "step 2: trim H.264 sources to ${CUT_SECONDS}s (-c copy)" +for out in "${!H264_TARGETS[@]}"; do + src="${H264_TARGETS[$out]}" + if file_ok "$out"; then + log " $out: present" + continue + fi + log " $out: trim from $src" + ffmpeg -hide_banner -loglevel error -y \ + -ss 0 -t "$CUT_SECONDS" -i "$src" \ + -c copy -avoid_negative_ts make_zero "$out.tmp.mp4" + mv "$out.tmp.mp4" "$out" +done + +# Step 3: HEVC re-encode each H.264 cut. ``-tag:v hvc1`` writes +# the iOS-friendly codec tag (matches what the asset processor +# emits at upload time -- keeps the cross-board fleet sha256 test +# meaningful). CRF defaults to 23 to roughly match the source's +# perceived quality so a passthrough vs re-encode A/B comparison +# isn't muddied by a visible quality delta. +declare -A HEVC_TARGETS=( + [bbb_1080p_30fps_hevc.mp4]=bbb_1080p_30fps.mp4 + [bbb_1080p_60fps_hevc.mp4]=bbb_1080p_60fps.mp4 + [bbb_4k_30fps_hevc.mp4]=bbb_4k_30fps.mp4 + [bbb_4k_60fps_hevc.mp4]=bbb_4k_60fps.mp4 +) + +log "step 3: HEVC re-encode (libx265 preset=medium crf=$HEVC_CRF)" +for out in "${!HEVC_TARGETS[@]}"; do + src="${HEVC_TARGETS[$out]}" + if file_ok "$out"; then + log " $out: present" + continue + fi + log " $out: encode from $src (this can take a minute or two)" + ffmpeg -hide_banner -loglevel error -y \ + -i "$src" \ + -c:v libx265 -preset medium -crf "$HEVC_CRF" -tag:v hvc1 \ + -c:a copy "$out.tmp.mp4" + mv "$out.tmp.mp4" "$out" +done + +# Step 4: report. +log "step 4: pack summary" +printf '\n%-32s %-10s %-12s %-10s %-8s\n' \ + file codec resolution fps duration_s +printf '%-32s %-10s %-12s %-10s %-8s\n' \ + -------- ----- ---------- --- --- +for f in bbb_1080p_30fps.mp4 bbb_1080p_60fps.mp4 \ + bbb_4k_30fps.mp4 bbb_4k_60fps.mp4 \ + bbb_1080p_30fps_hevc.mp4 bbb_1080p_60fps_hevc.mp4 \ + bbb_4k_30fps_hevc.mp4 bbb_4k_60fps_hevc.mp4; do + [[ -f "$f" ]] || continue + codec=$(ffprobe -v error -select_streams v:0 \ + -show_entries stream=codec_name -of csv=p=0 "$f") + width=$(ffprobe -v error -select_streams v:0 \ + -show_entries stream=width -of csv=p=0 "$f") + height=$(ffprobe -v error -select_streams v:0 \ + -show_entries stream=height -of csv=p=0 "$f") + fps_rat=$(ffprobe -v error -select_streams v:0 \ + -show_entries stream=r_frame_rate -of csv=p=0 "$f") + fps=$(echo "$fps_rat" | awk -F/ '{ printf "%.0f", $1/$2 }') + dur=$(ffprobe -v error \ + -show_entries format=duration -of csv=p=0 "$f") + printf '%-32s %-10s %-12s %-10s %-8s\n' \ + "$f" "$codec" "${width}x${height}" "$fps" \ + "$(printf '%.1f' "$dur")" +done + +log "done. pack lives at $DEST" diff --git a/bin/start_viewer.sh b/bin/start_viewer.sh index a83a6f5bd..a48cb3074 100755 --- a/bin/start_viewer.sh +++ b/bin/start_viewer.sh @@ -9,6 +9,49 @@ chgrp -f video /dev/vchiq chmod -f g+rwX /dev/vchiq +# Recreate the kernel's ``/dev/video-dec*`` symlinks inside the +# container for boards whose v4l2_request decoders are reachable +# from upstream mpv (RK3399 / Rock Pi 4 today; future Rockchip / +# Allwinner / Amlogic SBCs likely too). Privileged docker passes +# the underlying ``/dev/video*`` char devices through but mounts +# its own ``/dev`` tmpfs without the udev rules that produce the +# decoder symlinks on the host. ffmpeg's ``hevc_v4l2m2m`` / +# ``h264_v4l2m2m`` lookup expects ``/dev/video-dec*`` and dies +# with "Could not find a valid device" otherwise. +# +# We can't run udev inside the container (no privileged +# udevd, and /sys/class/video4linux is read-only via /sys +# bind), but we don't need to — the rule is mechanical: any +# /dev/video* whose /sys/class/video4linux//name reads as +# a stateless decoder driver gets a symlink. Iterate explicitly +# instead of shelling udev. +for dev_node in /dev/video*; do + [ -c "$dev_node" ] || continue + base=$(basename "$dev_node") + drv_name_file="/sys/class/video4linux/$base/name" + [ -r "$drv_name_file" ] || continue + name=$(cat "$drv_name_file" 2>/dev/null) + # Rockchip / Allwinner / Amlogic stateless decoders. The + # canonical kernel naming is: + # + # * ``rkvdec`` — Rock Pi 4's RK3399 HEVC + VP9 stateless + # decoder (and equivalents on RK3328 / RK356x / RK3588); + # * ``rockchip,-vpu-dec`` — the legacy "VPU" H.264 / + # MPEG block, exposed as a separate v4l2 node; + # * ``hantro-vpu`` / ``hantro-g*`` — same silicon family, + # different vendor-tree naming on a handful of boards; + # * ``cedrus`` — Allwinner H6 / H616 stateless decoder. + # + # We match the suffix ``-dec`` plus the ``rkvdec`` / ``cedrus`` + # / ``hantro`` prefixes so all the above hit the same alias + # rule without us enumerating every kernel build's exact name. + case "$name" in + rkvdec*|cedrus*|hantro*|*-vpu-dec|*-dec) + ln -snf "$dev_node" "/dev/video-dec${base#video}" + ;; + esac +done + # Set permission for sha file chown -f viewer /dev/snd/* chown -f viewer /data/.anthias/latest_anthias_sha @@ -116,36 +159,44 @@ fi # we just set; -E alone is subject to env_check / env_delete and is not # guaranteed for XDG_* on Debian's default sudoers. # -# x86 boards run under `cage`, a kiosk wlroots compositor, because -# balenaOS x86 doesn't expose /dev/fb0 (Qt's linuxfb plugin has nothing -# to draw to) and there's no host display server. cage acquires DRM -# master as root, exports WAYLAND_DISPLAY for its child, and exits when -# the child exits — so the existing kill -0 watchdog below still works. -# The inner sudo drops back to the viewer user; WAYLAND_DISPLAY has to -# be added to --preserve-env to survive sudo's env scrub. -if [ "$DEVICE_TYPE" = "x86" ] || [ "$DEVICE_TYPE" = "arm64" ]; then - # /dev/dri/renderD128 carries the host's `render` group, whose - # numeric GID is distro-dependent (typically 992 on Debian/Ubuntu, - # 109 elsewhere) and not present in the container's /etc/group. - # Without membership the `viewer` user can open card0 (group - # `video`, GID 44 — already a member) but not the render node, and - # VAAPI silently fails with "wayland: failed to open - # /dev/dri/renderD128". mpv then falls back to software decode and - # frames drop at 1080p on entry-level x86. Mirror the host GID - # into the container as a synthetic `host-render` group and add - # `viewer` to it, so the supplementary group list `sudo -u viewer` - # later resolves from /etc/group already includes render access. - if [ -e /dev/dri/renderD128 ]; then - render_gid=$(stat -c %g /dev/dri/renderD128) - if [ "$render_gid" -ne 0 ]; then - if ! getent group "$render_gid" >/dev/null; then - groupadd -g "$render_gid" host-render - fi - host_render_group=$(getent group "$render_gid" | cut -d: -f1) - usermod -aG "$host_render_group" viewer +# /dev/dri/renderD128 carries the host's `render` group, whose +# numeric GID is distro-dependent (typically 992 on Debian/Ubuntu, +# 109 elsewhere, 106 on Pi OS Bookworm) and not always present in +# the container's /etc/group. Without membership the `viewer` user +# can open card0 (group `video`, GID 44 — already a member) but +# not the render node. mpv uses the render node for --vo=gpu on +# every Qt 6 board, whether via wayland (cage path: x86 / arm64 / +# pi5) or drm (linuxfb path: pi4-64). Mirror the host GID into +# the container as a synthetic `host-render` group and add +# `viewer` to it; the supplementary group list `sudo -u viewer` +# later resolves from /etc/group then includes render access. +if [ -e /dev/dri/renderD128 ]; then + render_gid=$(stat -c %g /dev/dri/renderD128) + if [ "$render_gid" -ne 0 ]; then + if ! getent group "$render_gid" >/dev/null; then + groupadd -g "$render_gid" host-render fi + host_render_group=$(getent group "$render_gid" | cut -d: -f1) + usermod -aG "$host_render_group" viewer fi +fi +# x86 / arm64 / pi5 run under `cage`, a kiosk wlroots compositor. +# cage acquires DRM master as root, exports WAYLAND_DISPLAY for its +# child, and exits when the child exits — so the existing kill -0 +# watchdog below still works. The inner sudo drops back to the +# viewer user; WAYLAND_DISPLAY has to be added to --preserve-env to +# survive sudo's env scrub. +# +# Pi 4 falls through to the legacy direct-sudo path that runs under +# QT_QPA_PLATFORM=linuxfb. The V3D 6.0 doesn't have the bandwidth +# to composite cage on top of video at 4K (738 vo drops/30 s under +# cage vs 3-6 on the linuxfb + --gpu-context=drm path), so Pi 4 +# stays on linuxfb until either a newer mpv with v4l2request hwdec +# or a future Pi platform lets us re-evaluate. Qt5 boards (pi2/pi3) +# share the same direct-sudo fallback path. +case "$DEVICE_TYPE" in + x86|arm64|pi5) # libseat's default `logind` backend D-Buses into systemd-logind to # acquire a session, but containers have no logind session — cage # exits with "Could not get primary session for user". Switch to @@ -155,23 +206,40 @@ if [ "$DEVICE_TYPE" = "x86" ] || [ "$DEVICE_TYPE" = "arm64" ]; then # devices — a digital-signage kiosk has no keyboard or mouse. export LIBSEAT_BACKEND=builtin export WLR_LIBINPUT_NO_DEVICES=1 + + # cage default `-m extend` spans all enumerated DRM outputs, + # including ones that are physically disconnected — so a Pi user + # who plugs into the second micro-HDMI port (HDMI-A-2 instead of + # HDMI-A-1) ends up with cage rendering to a portion of the + # virtual canvas that lands on the disconnected connector, and a + # black screen. Trixie ships cage 0.1.x which has no `-o + # ` flag, but `-m last` restricts output to whichever + # connector came up most recently — for the boot-time case + # (which the kernel detects in enumeration order) that's the + # last connected output rather than the first. Good enough for + # the single-display kiosk path; dual-head signage is a separate + # workflow. + cage_mode=(-m last) + # cage runs as root (Dockerfile's USER root) and creates the # Wayland socket with root:root 0600 perms, so `sudo -u viewer` # below can't connect (Qt: "Failed to create wl_display # (Permission denied)"). Chown the socket to viewer in cage's # child *before* dropping privileges. cage exports WAYLAND_DISPLAY # before exec'ing the child, so the path is fully resolved here. - cage -- bash -c ' + cage "${cage_mode[@]}" -- bash -c ' chown viewer "${XDG_RUNTIME_DIR}/${WAYLAND_DISPLAY}" 2>/dev/null || true exec sudo \ --preserve-env=XDG_RUNTIME_DIR,QT_SCALE_FACTOR,PYTHONPATH,WAYLAND_DISPLAY,LANG,LANGUAGE,LC_ALL \ -E -u viewer \ dbus-run-session /venv/bin/python -m anthias_viewer ' & -else + ;; + *) sudo --preserve-env=XDG_RUNTIME_DIR,QT_SCALE_FACTOR,PYTHONPATH,LANG,LANGUAGE,LC_ALL -E -u viewer \ dbus-run-session /venv/bin/python -m anthias_viewer & -fi + ;; +esac # Wait for the viewer while true; do diff --git a/bin/upgrade_containers.sh b/bin/upgrade_containers.sh index fea09d9a0..1f22048ae 100755 --- a/bin/upgrade_containers.sh +++ b/bin/upgrade_containers.sh @@ -15,6 +15,12 @@ SCRIPT_DIR="$(cd "$(dirname "$0")" && pwd)" TOTAL_MEMORY_KB=$(grep MemTotal /proc/meminfo | awk {'print $2'}) export VIEWER_MEMORY_LIMIT_KB=$(echo "$TOTAL_MEMORY_KB" \* 0.8 | bc) export SHM_SIZE_KB="$(echo "$TOTAL_MEMORY_KB" \* 0.3 | bc | cut -d'.' -f1)" +# Memory cap for anthias-celery. 60% of host RAM is conservative +# headroom for the remaining celery workloads (ffprobe metadata, +# HEIC → WebP image conversion); the cap is here as a safety net +# against a decompression-bomb fixture or runaway ffprobe, not +# because routine workloads come anywhere near it. +export CELERY_MEMORY_LIMIT_KB=$(echo "$TOTAL_MEMORY_KB * 0.6" | bc | cut -d'.' -f1) GIT_BRANCH="${GIT_BRANCH:-master}" MODE="${MODE:-pull}" diff --git a/docker-compose.balena.dev.yml.tmpl b/docker-compose.balena.dev.yml.tmpl index d087fc938..1a79e4817 100644 --- a/docker-compose.balena.dev.yml.tmpl +++ b/docker-compose.balena.dev.yml.tmpl @@ -52,8 +52,6 @@ services: # Runs on the same image as anthias-server with a CMD override. # See docker-compose.yml.tmpl for context on the merge. image: ghcr.io/screenly/anthias-server:${GIT_SHORT_HASH}-${BOARD} - # nice + ionice keep the upload-time transcode pipeline from - # starving the on-device viewer; see docker-compose.yml.tmpl. command: > nice -n 19 ionice -c 3 celery -A anthias_server.celery_tasks.celery worker -B -n worker@anthias diff --git a/docker-compose.balena.yml.tmpl b/docker-compose.balena.yml.tmpl index c8c277664..2e71481a6 100644 --- a/docker-compose.balena.yml.tmpl +++ b/docker-compose.balena.yml.tmpl @@ -49,8 +49,6 @@ services: # Runs on the same image as anthias-server with a CMD override. # See docker-compose.yml.tmpl for context on the merge. image: ghcr.io/screenly/anthias-server:${GIT_SHORT_HASH}-${BOARD} - # nice + ionice keep the upload-time transcode pipeline from - # starving the on-device viewer; see docker-compose.yml.tmpl. command: > nice -n 19 ionice -c 3 celery -A anthias_server.celery_tasks.celery worker -B -n worker@anthias diff --git a/docker-compose.dev.yml b/docker-compose.dev.yml index bfea4c4d3..e995df557 100644 --- a/docker-compose.dev.yml +++ b/docker-compose.dev.yml @@ -35,6 +35,7 @@ services: redis: condition: service_started command: > + nice -n 19 ionice -c 3 celery -A anthias_server.celery_tasks.celery worker -B -n worker@anthias --loglevel=info --scheduler celery.beat.Scheduler environment: diff --git a/docker-compose.yml.tmpl b/docker-compose.yml.tmpl index 4ecd2925f..283ea0b20 100644 --- a/docker-compose.yml.tmpl +++ b/docker-compose.yml.tmpl @@ -61,6 +61,21 @@ services: - LC_ALL=${LC_ALL} privileged: true restart: always + # cage (the wlroots kiosk compositor used on Pi 5 / x86 / + # arm64 viewers) doesn't always release the DRM master on + # SIGTERM — its event loop blocks on a libinput shutdown + # path that hangs on certain kernels, and the kernel's GPU + # driver leaves dangling references that prevent the next + # container start from acquiring DRM master. ``stop_grace_period`` + # caps how long ``docker compose down/restart`` waits before + # SIGKILLing the container; 3s is short enough that an + # upgrade rolls in a reasonable time even if cage hangs. + # ``stop_signal: SIGKILL`` skips the SIGTERM-then-wait dance + # entirely on intentional stops — same effect as a kernel + # OOM killer landing on the viewer, which the next ``up`` + # cleans up from automatically. + stop_grace_period: 3s + stop_signal: SIGKILL shm_size: ${SHM_SIZE_KB}kb volumes: - resin-data:/data @@ -83,15 +98,18 @@ services: # and a separate celery image was duplicating ~825 MB extracted of # identical content per device. See refactor: drop celery image. image: ghcr.io/screenly/anthias-server:${DOCKER_TAG}-${DEVICE_TYPE} - # ``nice -n 19 ionice -c 3`` lowers CPU and IO priority to the - # idle class so the upload-time normalisation pipeline (HEIC→WebP, - # exotic-codec → board-appropriate H.264/HEVC transcode) never starves the on-device - # viewer mid-playback. Both wrappers are no-ops when the system is - # idle — a background sweep on a quiet device still runs at - # full speed; only contention with the viewer process slows it - # down. Subprocesses (ffmpeg, ffprobe, Pillow's libheif binding) - # inherit the wrapper's settings, so a single configuration here - # covers every workload the worker spawns. + # ``mem_limit`` keeps a runaway celery task (e.g. a HEIC → + # WebP convert on a decompression-bomb fixture) from eating + # the box's RAM and starving the viewer. 60% of host (computed + # in bin/upgrade_containers.sh) leaves comfortable headroom on + # every supported SBC. + mem_limit: ${CELERY_MEMORY_LIMIT_KB}k + # ``nice -n 19 ionice -c 3`` keeps any subprocess celery + # spawns (ffprobe, Pillow's libheif binding) at idle priority + # so the on-device viewer always wins CPU + IO contention. No + # current task is CPU-bound enough that this matters in + # practice, but the wrappers are cheap insurance against a + # pathological input. command: > nice -n 19 ionice -c 3 celery -A anthias_server.celery_tasks.celery worker -B -n worker@anthias diff --git a/docker/Dockerfile.viewer.j2 b/docker/Dockerfile.viewer.j2 index a90b399b6..cbe92caf3 100644 --- a/docker/Dockerfile.viewer.j2 +++ b/docker/Dockerfile.viewer.j2 @@ -39,6 +39,8 @@ RUN mkdir -p /out/bin /out/share/AnthiasWebview && \ {% include 'Dockerfile.base.j2' %} +{% include '_rpt1-ffmpeg-pin.j2' %} + {% if disable_cache_mounts %} RUN \ {% else %} @@ -68,15 +70,16 @@ COPY --from=webview-builder /out/bin/AnthiasWebview /usr/local/bin/AnthiasWebvie COPY --from=webview-builder /out/share/AnthiasWebview /usr/local/share/AnthiasWebview ENV QT_QPA_EGLFS_FORCE888=1 -{% if board in ('x86', 'arm64') %} -# balenaOS x86 doesn't expose /dev/fb0, so the linuxfb plugin has -# nothing to draw to. Run Qt under Wayland instead; bin/start_viewer.sh -# wraps the viewer in `cage`, a kiosk Wayland compositor that talks to -# KMS via wlroots. The same path covers arm64 boards (Armbian -# on Rock Pi / Orange Pi / Banana Pi, …), which similarly boot Mesa -# straight to DRM/KMS without a fbdev emulation node. +{% if board in ('x86', 'arm64', 'pi5') %} +# x86 / arm64 / pi5 run the viewer under `cage`, a kiosk wlroots +# compositor that talks straight to KMS. bin/start_viewer.sh wraps +# the viewer in cage; this env var tells Qt to load the qt6-wayland +# platform plugin so it renders into cage's surface. ENV QT_QPA_PLATFORM=wayland {% else %} +# Pi 4 (Qt6) and Pi 2 / Pi 3 (Qt5) keep the classic linuxfb path — +# the Pi 4 V3D 6.0 can't keep up with cage's composite pass on top +# of video, so the viewer renders directly to the framebuffer. ENV QT_QPA_PLATFORM=linuxfb {% endif %} diff --git a/docker/_rpt1-ffmpeg-pin.j2 b/docker/_rpt1-ffmpeg-pin.j2 new file mode 100644 index 000000000..99fd2cc4f --- /dev/null +++ b/docker/_rpt1-ffmpeg-pin.j2 @@ -0,0 +1,72 @@ +{% if board in ('pi4-64', 'pi5', 'arm64') %} +# archive.raspberrypi.com ships ffmpeg patched with v4l2_request +# enabled (``--enable-v4l2-request --enable-libudev +# --enable-vout-drm --enable-sand --enable-neon``). Stock Debian +# Trixie's ffmpeg drops v4l2_request entirely, so the stateless +# hardware decoders Anthias needs are unreachable without these +# packages: +# +# * Pi 4-64 — V3D V4L2 M2M (H.264 via stateful v4l2m2m, works on +# stock too) plus the dedicated rpi-hevc-dec block (HEVC via +# v4l2_request, needs ``+rpt1``). +# * Pi 5 — Hantro G2 HEVC via v4l2_request (needs ``+rpt1``). +# * arm64 — Rock Pi 4 / RK3399's ``rkvdec`` (HEVC) and Hantro VPU +# (H.264) are also stateless v4l2_request decoders. Live test +# on the Rock Pi 4 confirmed stock Debian arm64 ffmpeg can't +# reach them; the ``+rpt1`` build has the v4l2_request code +# paths that DO reach them. +# +# x86 stays on stock Debian — VAAPI already works there and Pi's +# repo doesn't add anything relevant for Intel / AMD iGPUs. +# +# Included by BOTH Dockerfile.viewer.j2 (for mpv's runtime decode) +# AND Dockerfile.server.j2 (for the asset processor walker's +# transcode pipeline — ``processing._decode_hwaccel_args`` emits +# ``-hwaccel drm`` which only works against the ``+rpt1`` ffmpeg). +# +# Apt pinning restricts the Pi repo to the mpv + FFmpeg library +# family only. Without this, Pi's curl, ca-certificates, etc. +# would silently shadow Debian's versions and we'd drift away +# from the rest of the image's apt baseline. +{% if disable_cache_mounts %} +RUN \ +{% else %} +RUN --mount=type=cache,target=/var/cache/apt,sharing=locked \ +{% endif %} + set -o pipefail; \ + apt-get update && \ + apt-get install -y --no-install-recommends ca-certificates curl && \ + install -d -m 0755 /etc/apt/keyrings && \ + curl -fsSL --retry 3 \ + https://archive.raspberrypi.com/debian/pool/main/r/raspberrypi-archive-keyring/raspberrypi-archive-keyring_2025.1+rpt1_all.deb \ + -o /tmp/raspberrypi-archive-keyring.deb && \ + dpkg-deb -x /tmp/raspberrypi-archive-keyring.deb /tmp/raspberrypi-keyring && \ + install -m 0644 \ + /tmp/raspberrypi-keyring/usr/share/keyrings/raspberrypi-archive-keyring.pgp \ + /etc/apt/keyrings/raspberrypi-archive.pgp && \ + rm -rf /tmp/raspberrypi-archive-keyring.deb /tmp/raspberrypi-keyring && \ + echo "deb [signed-by=/etc/apt/keyrings/raspberrypi-archive.pgp] https://archive.raspberrypi.com/debian/ trixie main" \ + > /etc/apt/sources.list.d/raspberrypi.list && \ + printf '%s\n' \ + 'Package: mpv libavcodec* libavdevice* libavfilter* libavformat* libavutil* libpostproc* libswresample* libswscale* ffmpeg' \ + 'Pin: origin archive.raspberrypi.com' \ + 'Pin-Priority: 1001' \ + '' \ + 'Package: *' \ + 'Pin: origin archive.raspberrypi.com' \ + 'Pin-Priority: 100' \ + > /etc/apt/preferences.d/raspberrypi +# The bare ``raspberrypi.gpg.key`` ASCII key on the Pi website still +# carries SHA1 binding signatures (RSA 2048, generated 2012-06-17) +# that Trixie's sqv refuses to certify under its post-2026-02-01 +# crypto policy ("Signing key … is not bound: No binding signature +# at time …, SHA1 is not considered secure since 2026-02-01"). The +# *deb-packaged* keyring ``raspberrypi-archive-keyring_*.pgp`` ships +# the same key fingerprint but with rebuilt binding signatures that +# sqv accepts — that's the keyring Pi OS Trixie itself installs and +# the reason ``apt update`` works on a real Pi 5 today. Fetching it +# directly here gives us the same verified path without depending +# on Pi's apt repo being reachable for an apt install (chicken-and- +# egg). Pin to a known-good version so a regenerated keyring deb +# can't silently change what we trust. +{% endif %} diff --git a/docs/board-enablement.md b/docs/board-enablement.md new file mode 100644 index 000000000..f05a8299b --- /dev/null +++ b/docs/board-enablement.md @@ -0,0 +1,194 @@ +# Board Enablement Test Bed + +A reproducible playback test bed for validating the viewer stack across boards. +Use this when changing anything that touches mpv flags, hwdec, the cage/Wayland +stack, or `src/anthias_viewer/media_player.py`. + +The viewer is tuned per-board (see `media_player.py` and `bin/start_viewer.sh`): + +| Device | Qt platform | Compositor | mpv VO | +|----------|-------------|-----------|---------------------------------| +| Pi 2 / 3 | Qt5 linuxfb | none | VLC | +| Pi 4-64 | Qt6 linuxfb | none | `--vo=gpu --gpu-context=drm` | +| Pi 5 | Qt6 wayland | cage | `--vo=gpu --gpu-context=wayland` | +| arm64 | Qt6 wayland | cage | `--vo=gpu --gpu-context=wayland` | +| x86 | Qt6 wayland | cage | `--vo=gpu --gpu-context=wayland` | + +Each combination has different hwdec, scaling, and compositing characteristics, +so a regression on one board can hide behind a clean run on another. Run the +same asset rotation everywhere and compare drop counts. + +## Goal: hardware-accelerated playback on every board + +Every clip Anthias displays should decode in hardware on the target board. +Software decode produces drops, heats the SoC, and on low-end Pis can't keep up +at 1080p30, let alone 4K. + +Anthias does *not* re-encode video uploads on-device — see the asset processor +docstring (`src/anthias_server/processing.py`) for why. The viewer's per-board +mpv hwdec dispatch handles every codec a modern board can decode in hardware +(H.264, HEVC, plus VAAPI's wider set on x86). For codecs the board can't decode +(MPEG-2 DVD rips, MPEG-4 ASP DivX clips, AV1 outside x86, …), playback will +visibly stutter; the asset list surfaces `metadata['video_codec']` / +`metadata['video_width']` / `metadata['video_height']` / `metadata['video_fps']` +so operators can spot a misfit clip before pushing it to the field. + +The viewer (`src/anthias_viewer/media_player.py`) selects the correct +mpv hwdec per codec on the target board. On Pi 4 / Pi 5 the launcher ffprobes +the asset and passes `--hwdec=v4l2m2m-copy` for H.264 or `--hwdec=drm-copy` +for HEVC directly on the mpv command line. (An earlier attempt used a Lua +`on_load` hook, but `video-codec-name` is empty at every script event before +hwdec init, so the hook was a silent no-op and `--hwdec=auto-copy` leaked +through; `auto-copy`'s upstream whitelist excludes `v4l2m2m-copy`, so H.264 fell +back to software.) + +## Hardware decode capabilities per Pi + +What the SoC can do, regardless of player: + +| Pi | SoC | H.264 HW | HEVC HW | VP9 / AV1 HW | +|------|----------|---------------------------|------------------------|--------------| +| 2 | BCM2836 | yes, up to 1080p (V3D IV) | **no** — no HEVC block | no | +| 3 | BCM2837 | yes, up to 1080p (V3D IV) | **no** — no HEVC block | no | +| 4 | BCM2711 | yes, up to 1080p60 (V3D 6.0 V4L2 M2M); 4K H.264 is past the V3D's envelope | yes, up to 4Kp60 (dedicated HEVC block, exposed as `v4l2_request_hevc`) | no | +| 5 | BCM2712 | yes in silicon (Hantro G1), but **not reachable through mpv** — no `v4l2-request` H.264 hwdec exists upstream | yes, up to 4Kp60 (Hantro G2, exposed as `v4l2_request_hevc`) | no | + +HEVC HW decode arrived with the Pi 4. Pi 2 / Pi 3 cannot decode HEVC in +hardware at all, and software HEVC on a Cortex-A53 won't even hit 1080p30 — so +uploading an HEVC asset for a Pi 2 / Pi 3 fleet member will play (badly) on the +SoC's software fallback. If you need HEVC content on that fleet, transcode it +upstream of the upload. + +> **Pi 5 4K HEVC requires `dtoverlay=vc4-kms-v3d,cma-512`.** The Hantro G2 +> driver allocates DMA buffers from the kernel's Contiguous Memory Allocator. +> Pi OS for Pi 5 reserves only 64 MB CMA by default (vs. 512 MB on Pi 4), +> which is enough for 1080p HEVC reference + output buffers but not 4K — at +> 4K mpv hits `v4l2_request_hevc_start_frame: Failed to get dst buffer` and +> silently SW-falls-back. Bumping `cma=512M` on the kernel cmdline does +> **not** work: the kernel takes the cmdline value over the device-tree +> `linux,cma` node, which leaves `rpi-hevc-dec` orphaned +> (`Failed to probe hardware -517`) and `/dev/video*` disappears entirely, +> killing HEVC HW at every resolution. The right fix is the +> `dtoverlay=vc4-kms-v3d,cma-512` line in `/boot/firmware/config.txt` — +> the vc4 overlay carries the `cma-N` knob and resizes the DT-declared +> region without orphaning the HEVC driver. The Anthias ansible template +> at `ansible/roles/system/templates/config.txt.j2` writes that line on +> install. + +## Rock Pi 4 / arm64 + +`bin/install.sh` sets `DEVICE_TYPE=arm64` for every aarch64 SBC it doesn't +recognise as a Pi. `anthias_host_agent` runs on the host and reads +`/proc/device-tree/model`; when it sees "Radxa ROCK Pi 4" it writes +`host:board_subtype = 'rockpi4'` to Redis. The viewer reads that key to +upgrade its `--hwdec=` choice from the catch-all `arm64` default to the +RK3399-specific `--hwdec=drm-copy` (v4l2_request, served by `rkvdec` for HEVC +and the Hantro VPU for H.264). + +The arm64 viewer image pulls `ffmpeg` and the libav* family from +`archive.raspberrypi.com` (the `+rpt1` build), which adds +`--enable-v4l2-request --enable-libudev --enable-vout-drm` — the same package +family Pi 4 / Pi 5 use, so the RK3399's stateless decoders are reachable via +the same mpv flag. The `start_viewer.sh` entrypoint creates the `/dev/video-dec*` +symlinks the v4l2_request decoder discovery code expects (privileged docker +mounts its own /dev tmpfs without udev's symlinks). The `+rpt1` repo is pinned +to only override ffmpeg + libav* + mpv on arm64; Pi userspace baseline is +unaffected on every board. + +## Sample pack + +Run `bin/generate_board_enablement_testbed.sh` on a workstation +(not the device under test) to produce the 8-clip pack: + +```bash +bash bin/generate_board_enablement_testbed.sh ~/bbb-testbed +``` + +The script: + +1. Downloads four Big Buck Bunny H.264 + AAC sources (public-domain, + from `download.blender.org/demo/movies/BBB`) — skipped if already + present. +2. Trims each to 60 seconds via `-c copy` (instant, no re-encode) — + produces the H.264 half of the pack. +3. Re-encodes each cut with `libx265 -preset medium -crf 23 -tag:v hvc1` + — produces the HEVC half. +4. Prints a verification table (codec + resolution + fps + duration + from `ffprobe`). + +| File | Codec | Resolution | fps | +|----------------------------------|-------|------------|-----| +| `bbb_1080p_30fps.mp4` | H.264 | 1920×1080 | 30 | +| `bbb_1080p_60fps.mp4` | H.264 | 1920×1080 | 60 | +| `bbb_4k_30fps.mp4` | H.264 | 3840×2160 | 30 | +| `bbb_4k_60fps.mp4` | H.264 | 3840×2160 | 60 | +| `bbb_1080p_30fps_hevc.mp4` | HEVC | 1920×1080 | 30 | +| `bbb_1080p_60fps_hevc.mp4` | HEVC | 1920×1080 | 60 | +| `bbb_4k_30fps_hevc.mp4` | HEVC | 3840×2160 | 30 | +| `bbb_4k_60fps_hevc.mp4` | HEVC | 3840×2160 | 60 | + +60 seconds per clip is enough to capture mpv's `hwdec-current` banner and +read a stable `Dropped:` count, while keeping a full pack regen achievable +in a few minutes on a laptop. Pass `CUT_SECONDS=N` to the script to change +the per-clip length; pass `HEVC_CRF=N` to override the encoder's quality +target. + +The script is idempotent: clips that already exist (and pass an `ffprobe` +sanity check) are skipped on re-run. A power cycle mid-encode leaves the +temp file as `*.tmp.mp4`; the next invocation regenerates from scratch. + +## The rotation + +Upload all eight files (4 × H.264 + 4 × HEVC) as Anthias assets and schedule +them back-to-back. Per-asset boundaries in the drop log make it easy to slice +results by resolution / fps / codec. + +## Drop logging + +Set `ANTHIAS_DEBUG_DROPS=1` on the `anthias-viewer` service (compose override +or `~/anthias/.env`). When enabled, `media_player.py`: + +- drops mpv's `--no-terminal`, so the status line + (`AV: 00:00:30 / ... Dropped: N`) is emitted continuously; +- redirects stdout/stderr to `/data/.anthias/mpv.log` inside the container, + which is `~/.anthias/mpv.log` on the host (or `~/.screenly/mpv.log` on + pre-rebrand installs); +- writes a `--- mpv launch ---` marker before each mpv launch so the + log can be sliced per asset; +- captures mpv's `hwdec-current` and VO init banners on stderr — confirms + `--vo=gpu --gpu-context=drm` (Pi 4) vs `--gpu-context=wayland` (Pi 5 / x86 + / arm64) actually took effect, and confirms which hwdec the per-codec + dispatch selected. + +With the env var unset, the viewer keeps its silent `DEVNULL` behaviour — +no host-side log file. + +## Reading the log + +`Dropped:` in mpv's status line is cumulative for a single mpv process, and +the viewer spawns one process per asset, so the last `Dropped:` before the +next `--- mpv launch` marker is that asset's final count over its playback +window. + +```bash +grep -E "^--- mpv launch|Dropped:" ~/.anthias/mpv.log | tail -80 +``` + +For a single rolling sample on a running device: + +```bash +tail -F ~/.anthias/mpv.log | grep --line-buffered -E "launch|Dropped:|hwdec-current|VO:" +``` + +## Reporting + +When attaching results to a PR, include: + +- board + Qt/compositor combination (one row of the table above); +- one drop count per asset, taken from the last `Dropped:N` of each asset's + window; +- the matching `VO:` / `hwdec-current` banner lines so the run can be tied to + a specific stack. + +For comparable numbers, let the rotation play at least two full cycles before +sampling — first cycle includes asset-cache warmup and webview teardown. diff --git a/src/anthias_common/board.py b/src/anthias_common/board.py new file mode 100644 index 000000000..8926a5b89 --- /dev/null +++ b/src/anthias_common/board.py @@ -0,0 +1,67 @@ +"""Runtime board identification. + +``bin/install.sh`` writes a coarse ``DEVICE_TYPE`` env var on every +device (``pi4-64``, ``pi5``, ``x86``, or the catch-all ``arm64`` / +``generic-arm64`` for every aarch64 SBC it doesn't recognise as a +Pi). For SoCs whose silicon offers HW decode that the catch-all +image can address (Rock Pi 4 → RK3399 via v4l2_request), the +``anthias_host_agent`` process publishes a more specific subtype to +Redis at ``host:board_subtype`` (e.g. ``'rockpi4'``) by reading +``/proc/device-tree/model`` on the host. + +Both the server's asset processor (deciding whether to accept a +codec) and the viewer's hwdec dispatch (deciding which mpv +``--hwdec=`` value to ask for) need the same upgraded key. This +module owns the resolution so the two sides don't drift. +""" + +from __future__ import annotations + +import os + +from anthias_common.utils import connect_to_redis + +# DEVICE_TYPE values that trigger the host_agent subtype lookup. +# ``generic-arm64`` is the legacy label that pre-rename arm64 images +# still carry; both share the same Rock-Pi-via-host_agent upgrade +# path. +ARM64_DEVICE_TYPES = frozenset({'arm64', 'generic-arm64'}) + + +def get_board_subtype() -> str | None: + """Return the host_agent-published board subtype, or ``None``. + + Any failure (Redis down, key missing, decode error) returns + ``None`` so the caller falls back to the raw DEVICE_TYPE. + """ + try: + r = connect_to_redis() + value = r.get('host:board_subtype') + except Exception: + return None + if isinstance(value, bytes): + try: + decoded = value.decode('utf-8') + except UnicodeDecodeError: + return None + return decoded.strip().lower() or None + if isinstance(value, str): + return value.strip().lower() or None + return None + + +def resolve_device_key() -> str: + """Return ``DEVICE_TYPE`` upgraded via the host_agent's published + board subtype when applicable. + + Lowercased and whitespace-stripped for stable dict lookup. An + unset or unrecognised ``DEVICE_TYPE`` returns an empty string — + callers treat that as "no HW path known on this device" rather + than guessing. + """ + key = os.environ.get('DEVICE_TYPE', '').strip().lower() + if key in ARM64_DEVICE_TYPES: + sub = get_board_subtype() + if sub: + return sub + return key diff --git a/src/anthias_host_agent/__main__.py b/src/anthias_host_agent/__main__.py index cdf16fec6..ba52f90df 100755 --- a/src/anthias_host_agent/__main__.py +++ b/src/anthias_host_agent/__main__.py @@ -142,6 +142,69 @@ def process_message(message: dict[str, Any]) -> None: logging.info('Received unsolicited message: %s', message) +def detect_board_subtype() -> str | None: + """Identify the SBC by reading ``/proc/device-tree/model``. + + Returns a stable short token (e.g. ``'rockpi4'``) when the model + string matches a known board, or ``None`` for unknown boards / + hosts without a device tree. The viewer reads the value the + publisher writes (``host:board_subtype``) to pick the right + ``--hwdec=`` for the SoC. + + Anthias's ``bin/install.sh`` writes ``DEVICE_TYPE=arm64`` for + every aarch64 SBC it doesn't recognise as a Pi. Most such boards + have no upstream-mpv HW decode path, but a few (Rock Pi 4 with + RK3399's Hantro VPU via v4l2m2m) do. Knowing which is which at + runtime lets the viewer pick the right ``--hwdec=`` value + without forcing operators to manually distinguish images. + + Host_agent runs on the host (not in a container) so it can + read the device tree directly — the alternative (mounting + ``/proc/device-tree`` into every container) is heavier and + doesn't compose well with balena. + """ + try: + with open('/proc/device-tree/model', 'rb') as f: + # Kernel writes a null-terminated UTF-8 string. + model = f.read().decode('utf-8', 'replace').strip('\x00 \n\t') + except OSError: + return None + if not model: + return None + model_low = model.lower() + # "Radxa ROCK Pi 4B" (and 4A / 4C variants — all RK3399). + if 'rock pi 4' in model_low: + return 'rockpi4' + return None + + +def set_board_subtype(rdb: 'redis.Redis') -> None: + """Publish the host's board subtype to Redis. + + Server + viewer read ``host:board_subtype`` to upgrade the + catch-all ``arm64`` DEVICE_TYPE into a board-specific matrix + key when one is detected. Written before + ``host_agent_ready`` flips so consumers don't read a stale + None when they wait on the readiness flag. + + On an unknown board (or a host without a device tree) the key + is set to the empty string. The server-side reader treats + empty / missing identically — falls back to the static + DEVICE_TYPE matrix entry — but writing the empty string still + distinguishes "host_agent ran and didn't recognise this board" + from "host_agent never ran". + """ + subtype = detect_board_subtype() or '' + rdb.set('host:board_subtype', subtype) + if subtype: + logging.info('Published board subtype %r to redis', subtype) + else: + logging.info( + 'No known board subtype for /proc/device-tree/model — ' + 'staying on DEVICE_TYPE-derived envelope' + ) + + def subscriber_loop() -> None: # On first boot the redis container may not yet accept connections; # retry quietly instead of crashing the unit on every attempt. @@ -162,6 +225,7 @@ def subscriber_loop() -> None: ) pubsub = rdb.pubsub(ignore_subscribe_messages=True) pubsub.subscribe(CHANNEL_NAME) + set_board_subtype(rdb) rdb.set('host_agent_ready', 'true') logging.info( 'Subscribed to channel %s, ready to process messages', CHANNEL_NAME diff --git a/src/anthias_server/api/serializers/v1_1.py b/src/anthias_server/api/serializers/v1_1.py index 13d3e7ae4..c28c6a162 100644 --- a/src/anthias_server/api/serializers/v1_1.py +++ b/src/anthias_server/api/serializers/v1_1.py @@ -185,15 +185,16 @@ def prepare_asset(self, data: dict[str, Any]) -> dict[str, Any]: ): raise Exception('Could not retrieve file. Check the asset URL.') - # Flag the freshly-uploaded local video for the normalisation - # pipeline. Fixes GH #2870: pre-fix, v1 / v1.1 left the file - # in whatever codec the operator uploaded (e.g. MPEG-1 on an - # x86 board with passthrough_video_codecs={'h264','hevc'}), - # and the viewer silently skipped it forever. Set - # ``is_processing=1`` so the viewer drops the in-flight row - # from rotation until ``normalize_video_asset`` finalises it. - # Image normalisation deliberately stays opt-in via v1.2 / v2 - # — see the class-level ``_pending_normalize`` docstring. + # Flag the freshly-uploaded local video for the + # normalisation pipeline. Fixes GH #2870: pre-fix, v1 / v1.1 + # left the row at ``is_processing=False`` and the viewer + # would try to play the upload before ffprobe metadata had + # been written, silently skipping anything mpv couldn't size. + # Set ``is_processing=1`` so the viewer drops the in-flight + # row from rotation until ``normalize_video_asset`` finalises + # it. Image normalisation deliberately stays opt-in via + # v1.2 / v2 — see the class-level ``_pending_normalize`` + # docstring. if ( is_local_upload and not is_youtube diff --git a/src/anthias_server/app/static/sass/_styles.scss b/src/anthias_server/app/static/sass/_styles.scss index 8b3739262..bb2185ef1 100644 --- a/src/anthias_server/app/static/sass/_styles.scss +++ b/src/anthias_server/app/static/sass/_styles.scss @@ -728,23 +728,91 @@ label { text-align: left; } } // Shown in place of the active toggle when the upload-time -// normalisation pipeline failed (corrupt HEIC, ffmpeg error, etc.). -// Same shape as .processing-pill so the column layout doesn't shift -// between in-progress / failed / done states; warn-coloured so a -// row that needs operator attention is visible at a glance. The -// hover tooltip on the element itself carries metadata.error_message -// from the row. +// normalisation pipeline failed (corrupt HEIC, ffmpeg error, +// unsupported video codec, ...). Same shape as .processing-pill so +// the column layout doesn't shift between in-progress / failed / +// done states; warn-coloured so a row that needs operator +// attention is visible at a glance. Rendered as a {% else %}
{% csrf_token %} diff --git a/src/anthias_server/app/views.py b/src/anthias_server/app/views.py index 114cdee38..587892433 100644 --- a/src/anthias_server/app/views.py +++ b/src/anthias_server/app/views.py @@ -459,6 +459,14 @@ def _safe_ext(candidate: str) -> str: play_order=play_order, start_date=now, end_date=now + timedelta(days=30), + # Stash the operator's original filename. The on-disk file + # is renamed to . at upload time (see + # ``final_name = uuid.uuid4().hex`` above) so the operator's + # local filename would otherwise be lost — but the video + # gate's ``UnsupportedVideoCodecError`` recipe wants to + # quote a name the operator can paste straight into their + # terminal, which is the upload name, not the on-disk UUID. + metadata={'upload_name': upload_name}, ) # Route through the shared ``dispatch_normalize_*`` helpers rather # than ``.delay()`` directly so the @@ -473,7 +481,7 @@ def _safe_ext(candidate: str) -> str: request, toast=( 'info', - f'Uploaded {upload_name} — analysing video…', + f'Uploaded {upload_name} — reading metadata…', ), ) if needs_image_normalize: diff --git a/src/anthias_server/celery_tasks.py b/src/anthias_server/celery_tasks.py index 1e927d18c..8a6c0b46a 100755 --- a/src/anthias_server/celery_tasks.py +++ b/src/anthias_server/celery_tasks.py @@ -9,10 +9,24 @@ import django import sh from celery import Celery, Task +from django.apps import apps as _django_apps from PIL import UnidentifiedImageError from tenacity import Retrying, stop_after_attempt, wait_fixed -django.setup() +# ``django.setup()`` is not reentrant — calling it while +# ``apps.populate()`` is still running (e.g. when an ``AppConfig.ready`` +# hook imports this module) raises ``RuntimeError: populate() isn't +# reentrant`` and the import dies, taking the caller down silently in +# any try/except chain. ``apps_ready`` flips to ``True`` after Django +# finishes the import phase but *before* the per-app ``ready`` hooks +# run, so the check correctly distinguishes: +# +# * standalone celery worker process → Django not initialised yet +# (``apps_ready=False``) → ``setup()`` runs as before; +# * import from inside an ``AppConfig.ready`` (server process) +# → Django is mid-populate (``apps_ready=True``) → skip. +if not _django_apps.apps_ready: + django.setup() # Place imports that uses Django in this block. @@ -68,10 +82,10 @@ RECONCILE_STUCK_INTERVAL_S = 60 * 10 # Age threshold for considering a row stuck. Has to be *longer* than -# the longest reasonable Celery task: ``NORMALIZE_VIDEO_TIME_LIMIT_S`` -# is 30 min and ``YOUTUBE_DOWNLOAD_TIME_LIMIT_S`` is 15 min, so 60 min -# is a safe floor. A row past the threshold either had its worker -# time-limit expire (in which case ``on_failure`` should already have +# the longest reasonable Celery task: ``YOUTUBE_DOWNLOAD_TIME_LIMIT_S`` +# is 15 min, so 60 min is a safe floor. A row past the threshold +# either had its worker time-limit expire (in which case +# ``on_failure`` should already have # cleared the flag — and the reconciler only sees rows where it # didn't, e.g. SIGKILL before on_failure could run) OR was never # picked up at all (Redis flake during ``.delay()``, worker crashed @@ -217,20 +231,23 @@ def cleanup() -> None: # the pre-rebrand prefix (~/screenly_assets/..., now a symlink to # ~/anthias_assets) are recognized as live and their files aren't # mistaken for orphans on upgraded installs. + # asset_dir_real = path.realpath(asset_dir) - referenced = set() - for uri in ( - Asset.objects.exclude(uri__isnull=True) - .exclude(uri__exact='') - .values_list('uri', flat=True) - ): - if not uri: - continue + referenced: set[str] = set() + + def _claim(p: str | None) -> None: + if not p: + return try: - if path.realpath(path.dirname(uri)) == asset_dir_real: - referenced.add(path.basename(uri)) + if path.realpath(path.dirname(p)) == asset_dir_real: + referenced.add(path.basename(p)) except OSError: - continue + return + + for uri in Asset.objects.exclude(uri__isnull=True).values_list( + 'uri', flat=True + ): + _claim(uri) cutoff = 60 * 60 # match the .tmp guard above now = time.time() for entry in os.scandir(asset_dir): @@ -614,13 +631,9 @@ def download_youtube_asset(asset_id: str, uri: str) -> None: _notify(asset_id, reload_viewer=False) - # Hand off to the per-board normalisation pass. This is what - # gives YouTube downloads the same codec / container guarantees - # as direct file uploads: ffprobe → passthrough on H.264/HEVC - # (per board profile) or transcode to libx264/libx265 otherwise. - # It also writes ``original_ext`` / ``transcoded`` / - # ``transcode_target`` to metadata, so the operator's view of a - # YouTube row carries the same diagnostic shape as a file upload. + # Hand off to ``normalize_video_asset`` so the YouTube download + # gets the same ffprobe metadata pass (codec / dims / fps / + # duration written into ``metadata``) as a direct file upload. dispatch_normalize_video(asset_id) @@ -801,9 +814,9 @@ def reconcile_stuck_processing() -> None: ``RECONCILE_STUCK_THRESHOLD_S`` (60 min) from that stamp — at a 10-min sweep cadence that's six sweeps where the row stays inside the grace window before becoming eligible. The grace is - deliberately long enough to cover the worst-case live transcode - (``NORMALIZE_VIDEO_TIME_LIMIT_S=30min``) plus margin, so a - still-in-flight task never gets yanked out from under itself. + deliberately longer than any single task's ``time_limit`` (the + longest is the 15-min YouTube download) so a still-in-flight + task never gets yanked out from under itself. Rows older than ``RECONCILE_STUCK_THRESHOLD_S`` (60 min) get re-dispatched via the normalisation path matching their mimetype. @@ -998,8 +1011,6 @@ def revalidate_asset_url(asset_id: str) -> None: # the worker and ``apply_async`` works out of the box. from anthias_server import processing # noqa: E402 -NORMALIZE_VIDEO_TIME_LIMIT_S = processing.NORMALIZE_VIDEO_TIME_LIMIT_S - @celery.task( base=processing._NormalizeAssetTask, @@ -1046,7 +1057,7 @@ def normalize_image_asset(asset_id: str) -> None: @celery.task( base=processing._NormalizeAssetTask, - time_limit=NORMALIZE_VIDEO_TIME_LIMIT_S, + time_limit=120, autoretry_for=(OSError,), # Same rationale as normalize_image_asset above: a missing source # file is permanent and should land on on_failure right away. @@ -1057,22 +1068,15 @@ def normalize_image_asset(asset_id: str) -> None: max_retries=1, ) def normalize_video_asset(asset_id: str) -> None: - """Probe the upload; passthrough or transcode to a board-appropriate - codec in MP4. - - The output codec is decided by ``processing._resolve_board_profile``: - libx264 on legacy Pi 2/Pi 3 (mmal-vc4 path; no HEVC hardware) and - libx265 with the iOS-friendly ``-tag:v hvc1`` on Pi 4-64 / Pi 5 / - x86 (mpv path; HEVC hardware-decoded on Pi 4 / x86, software on - Pi 5). The on-device player only ever sees a codec it can decode. - - ffmpeg is wrapped with ``-threads 2`` so two cores stay free for - the on-device viewer; the celery worker itself runs under - ``nice -n 19 ionice -c 3`` (set in docker-compose.yml.tmpl). - - Retry policy mirrors ``download_youtube_asset``: OSError gets one - retry (transient IO), ffmpeg subprocess failures and timeouts are - permanent and land on on_failure. + """Probe the upload with ffprobe, write codec / dims / fps / + duration into ``metadata``, and clear ``is_processing``. The + asset file is never rewritten — see + ``processing._run_video_normalisation`` for why. + + Retry policy: OSError gets one retry (transient IO), an ffprobe + timeout or non-zero exit is permanent and lands on on_failure + via ``_NormalizeAssetTask``. ``time_limit=120`` is the worst-case + ffprobe wall-clock (``_FFPROBE_TIMEOUT_S`` is 60 s) doubled. """ asset = processing._row_or_none(asset_id) if asset is None: diff --git a/src/anthias_server/processing.py b/src/anthias_server/processing.py index 056d61b6d..aaf8869f6 100644 --- a/src/anthias_server/processing.py +++ b/src/anthias_server/processing.py @@ -11,13 +11,20 @@ (BMP especially). JPEG / PNG / WebP / GIF / SVG short-circuit through the no-op branch — they're already viewer-friendly *and* well-compressed. -* ``normalize_video_asset`` — probes the upload's container/codec with - ffprobe and either passes it through (rename only) or transcodes - with ffmpeg's ``-threads 2`` to a board-appropriate codec: libx264 - on legacy Pi 2/Pi 3 (mmal-vc4 path; no hardware HEVC) and libx265 - with the iOS-friendly ``hvc1`` tag on Pi 4-64 / Pi 5 / x86 (mpv - path; HEVC hardware-decoded on Pi 4 / x86, software on Pi 5). The - on-device player only ever sees a codec it can decode. +* ``normalize_video_asset`` — runs ffprobe on the upload and records + what it finds in ``metadata`` (codec, dimensions, fps, audio codec, + container, duration). The file itself is never rewritten. Anthias + does not transcode video on-device: the viewer's per-board mpv + hwdec dispatch already handles every codec a modern board can play + in hardware (H.264, HEVC, plus VAAPI's wider set on x86), and the + on-device libx265 / libx264 transcode path we tried in this PR's + earlier revisions wedged a Pi 4's celery worker for 99 minutes on a + single 4K60 H.264 → HEVC pass before zombieing. For codecs the + board genuinely can't decode (MPEG-2, MPEG-4 ASP, ...), playback + will stutter and the operator's recovery is to upload a transcoded + copy — the metadata fields surface what's on each row so the + operator can see the codec / dims / fps before pushing the asset to + the field. Both tasks follow the YouTube-download Celery pattern in ``anthias_server.celery_tasks``: @@ -25,20 +32,20 @@ * The upload-path serializer flips ``is_processing=True`` and enqueues the task before returning. The viewer treats in-flight rows as not-displayable and silently skips them during rotation. -* On success the task atomically replaces the file at the row's - ``uri``, refreshes the duration where applicable, writes - ``metadata['original_ext']`` / ``metadata['transcoded']`` / - ``metadata['converted']``, and clears ``is_processing``. +* On success the task writes the metadata fields and clears + ``is_processing``. The image task additionally rewrites the file + in place to WebP; the video task leaves the file unchanged. * On failure the row's ``metadata['error_message']`` is filled in and ``is_processing`` is cleared via the custom ``Task.on_failure`` hook so an operator can edit / delete the row instead of being stuck on the "Processing" pill forever. Tasks run inside the same ``anthias-celery`` worker that handles the -existing ``download_youtube_asset`` flow. The compose file wraps the -worker command with ``nice -n 19 ionice -c 3`` so a transcode never -starves the on-device viewer; the ffmpeg invocation here additionally -caps thread count to two cores. +existing ``download_youtube_asset`` flow. The compose file still +wraps the worker with ``nice -n 19 ionice -c 3`` and a memory limit; +those are defensive — none of the remaining task bodies are CPU-bound +now that the on-device video transcode is gone, but a Pillow decode +on a 100 MP TIFF can still pressure RAM on a 1 GB Pi 2. """ from __future__ import annotations @@ -46,6 +53,7 @@ import json import logging import os +import shlex from os import path from typing import Any @@ -53,218 +61,10 @@ from celery import Task from PIL import Image, UnidentifiedImageError -from anthias_common.utils import get_video_duration +from anthias_common.board import resolve_device_key from anthias_server.app.models import Asset -# Containers whose H.264/HEVC payloads play directly in mpv / VLC -# on the Pi without remuxing. Anything outside this set falls through -# to a full transcode regardless of codec — a "passthrough" rename -# preserving a weird container would still need a downstream remux -# to land in MP4, and the viewer's media stack is happiest on .mp4. -# Keeping the list explicit also stops a typo'd extension from being -# silently retained. -# -# The set carries BOTH the short extension labels (``ts``, ``mkv``, -# ``mpg``) AND the canonical ffprobe ``format_name`` tokens -# (``mpegts``, ``matroska``, ``mpeg``) because the same set is -# matched against two sources of truth: -# -# * the upload's filename extension (extension fallback path in -# ``_ffprobe_summary`` — short labels), and -# * ffprobe's reported ``format.format_name`` (canonical names). -# -# A pure short-label set would force unnecessary transcodes whenever -# ffprobe's name (e.g. ``mpegts`` for an MPEG-TS upload) didn't -# match the extension's short label (``ts``). Listing both keeps the -# decision aligned across detection paths. -_PASSTHROUGH_CONTAINERS = frozenset( - { - # Short extension labels (matched against filename ext). - 'mp4', - 'm4v', - 'mkv', - 'mov', - 'webm', - 'ts', - 'mpg', - 'mpeg', - 'flv', - 'avi', - # ffprobe ``format_name`` tokens not already covered above. - # ``matroska`` for .mkv (ffprobe reports ``matroska,webm``). - # ``mpegts`` for .ts (ffprobe reports ``mpegts`` not ``ts``). - # ``mov`` / ``mp4`` / ``mpeg`` / ``flv`` / ``avi`` / ``webm`` - # are the canonical names *and* extension labels — only - # listed once above. - 'matroska', - 'mpegts', - } -) - - -# Audio codecs the viewer can demux without a transcode. ``None`` is -# represented as the literal string ``'none'`` so a probe result with -# no audio stream still falls in the "passthrough OK" set. -_PASSTHROUGH_AUDIO_CODECS = frozenset( - {'aac', 'mp3', 'opus', 'vorbis', 'ac3', 'none'} -) - - -# --------------------------------------------------------------------------- -# Per-board transcode profile -# --------------------------------------------------------------------------- -# -# The right "video codec" for an Anthias device depends on what the -# on-device player can hardware-decode (or software-decode at real -# time). The matrix this PR locks in: -# -# ┌──────────┬─────────────────┬──────────────┬──────────────┐ -# │ Board │ Player │ HEVC OK? │ Target codec │ -# ├──────────┼─────────────────┼──────────────┼──────────────┤ -# │ pi2/pi3 │ VLC + mmal-vc4 │ no │ H.264 │ -# │ pi4-64 │ mpv + V4L2 HEVC │ HW-decoded │ HEVC │ -# │ pi5 │ mpv + SW decode │ A76 SW @ 1080p │ HEVC │ -# │ x86 │ mpv + va/nv/qsv │ HW-decoded │ HEVC │ -# │ unset │ (dev / unknown) │ assume no │ H.264 │ -# └──────────┴─────────────────┴──────────────┴──────────────┘ -# -# Two reasons to actually emit HEVC instead of always-H.264: -# -# 1. Storage. Anthias devices have small SD cards / eMMC modules; an -# HEVC re-encode at equivalent visual quality is roughly 30–50% -# smaller than H.264. For a fleet rotating dozens of clips that -# compounds. -# 2. Decode load. Pi 5 has no hardware video decoder at all; the CPU -# handles every codec in software. HEVC's better compression at -# the same quality means fewer bits the decoder has to chew -# through, which trades coding-tool complexity for raw -# bandwidth — a wash on Pi 5 in practice, but never worse. -# -# The mapping keys match ``DEVICE_TYPE`` (set by the image builder in -# the Dockerfile, read at celery-task time via ``os.environ``) rather -# than the runtime-detected ``get_device_type()``. The celery worker -# shares the env var with anthias-server; it does NOT mount -# ``/proc/device-tree/model`` from the host. The image builder also -# uses these exact strings (``pi2`` / ``pi3`` / ``pi4-64`` / ``pi5`` / -# ``x86``), so a build-time decision and the transcode-time decision -# always agree. Fallback to ``_DEFAULT_PROFILE`` (H.264) when the env -# var is unset — keeps the dev-environment path safe and gives an -# unknown future board the most-compatible codec. -_BoardProfile = dict[str, Any] - - -# ffmpeg encoder args. Each list is what gets passed between ``-i -# `` and ```` for the video stream — audio always -# becomes AAC 192k via _AUDIO_TRANSCODE_ARGS. ``-tag:v hvc1`` on the -# HEVC encoder writes the iOS-friendly ``hvc1`` codec tag instead of -# ffmpeg's default ``hev1``; mpv/VLC handle either, but hvc1 is the -# broader-compat choice if we ever serve these files to a browser. -# -# CRF values are chosen to roughly match perceived quality across -# codecs: libx264 CRF 23 ≈ libx265 CRF 28. Both leave plenty of -# headroom for a fleet's typical image-and-text signage content. -_H264_VIDEO_ARGS = [ - '-c:v', - 'libx264', - '-preset', - 'medium', - '-crf', - '23', -] - -_HEVC_VIDEO_ARGS = [ - '-c:v', - 'libx265', - '-preset', - 'medium', - '-crf', - '28', - '-tag:v', - 'hvc1', -] - -_AUDIO_TRANSCODE_ARGS = ['-c:a', 'aac', '-b:a', '192k'] - - -_DEFAULT_PROFILE: _BoardProfile = { - # Default lands on H.264 — safe on every Anthias-supported device, - # and the fallback for ``DEVICE_TYPE`` unset (dev environment) or - # an unrecognised value. - 'transcode_target': 'h264', - 'passthrough_video_codecs': frozenset({'h264'}), - 'video_args': _H264_VIDEO_ARGS, -} - - -_BOARD_PROFILES: dict[str, _BoardProfile] = { - # Legacy 32-bit Pi boards: VLC + mmal-vc4 path. mmal hardware - # decode is H.264-only, the CPU is too slow to software-decode - # 1080p HEVC, so HEVC is *not* in the passthrough set — uploading - # an HEVC clip to a pi2/pi3 must go through a libx264 transcode. - 'pi2': { - 'transcode_target': 'h264', - 'passthrough_video_codecs': frozenset({'h264'}), - 'video_args': _H264_VIDEO_ARGS, - }, - 'pi3': { - 'transcode_target': 'h264', - 'passthrough_video_codecs': frozenset({'h264'}), - 'video_args': _H264_VIDEO_ARGS, - }, - # 64-bit Pi 4 with mpv + KMS (`--vo=drm`): the kernel's V4L2 - # stateful HEVC decoder driver (/dev/video10 family) is wired up - # and mpv's ``--hwdec=auto-safe`` selects ``v4l2request`` for - # hevc. Both H.264 and HEVC pass through. - 'pi4-64': { - 'transcode_target': 'hevc', - 'passthrough_video_codecs': frozenset({'h264', 'hevc'}), - 'video_args': _HEVC_VIDEO_ARGS, - }, - # Pi 5: no hardware video decoder block at all (RP1 dropped it - # vs. pi4). The Cortex-A76 quad-core software-decodes 1080p H.264 - # *and* 1080p HEVC at real time, so HEVC is fine. Picking HEVC - # also saves disk: a typical 5-minute clip is ~30% smaller after - # re-encode than the equivalent H.264 at perceptual parity. - 'pi5': { - 'transcode_target': 'hevc', - 'passthrough_video_codecs': frozenset({'h264', 'hevc'}), - 'video_args': _HEVC_VIDEO_ARGS, - }, - # x86: mpv + ``--hwdec=auto-safe`` selects vaapi (Intel/AMD), - # nvdec (NVIDIA), or qsv (Intel iGPU) and every modern x86 - # platform handles both H.264 and HEVC in hardware. Even on a - # software-decode-only x86 box, the CPU has plenty of headroom. - 'x86': { - 'transcode_target': 'hevc', - 'passthrough_video_codecs': frozenset({'h264', 'hevc'}), - 'video_args': _HEVC_VIDEO_ARGS, - }, -} - - -def _resolve_board_profile() -> _BoardProfile: - """Map the runtime ``DEVICE_TYPE`` env var to a transcode profile. - - The image builder writes ``DEVICE_TYPE=`` into the server - image's env at build time (see ``docker/Dockerfile.server.j2``); - the celery worker inherits the same env. Looking it up here means - a transcode pipeline running on a pi5 image always picks the pi5 - profile, even if the underlying CPU briefly looks different to - /proc inspection (Balena / dev workflows can run amd64 builds on - x86 hardware while still claiming a Pi target). - - Falls back to ``_DEFAULT_PROFILE`` (H.264) on: - * unset env var (host dev environment, ``ENVIRONMENT=test``), - * a future board name we haven't profiled yet. - - The H.264 default is the most compatible choice — every Anthias - device, present and historic, plays libx264. - """ - device_type = os.environ.get('DEVICE_TYPE', '').strip().lower() - return _BOARD_PROFILES.get(device_type, _DEFAULT_PROFILE) - - # Image extensions we route through the conversion task. The # motivation differs by format: # @@ -519,7 +319,9 @@ def _format_subprocess_stderr(exc: sh.ErrorReturnCode) -> str: return raw.decode('utf-8', errors='replace').strip() -def _set_processing_error(asset_id: str, message: str) -> None: +def _set_processing_error( + asset_id: str, message: str, recipe: str = '' +) -> None: """Persist a human-readable error and clear is_processing. Both tasks land here on a permanent failure (corrupt HEIC, @@ -528,6 +330,11 @@ def _set_processing_error(asset_id: str, message: str) -> None: instead of leaving the row stuck at ``is_processing=True`` is the contract called out by the issue's acceptance criteria. Operators surface the message via the v2 API's ``metadata`` field. + + ``recipe``, when present, persists alongside the message as + ``metadata.error_recipe`` — the dashboard renders it in a + copyable ```` block in the Edit Asset modal. Empty + ``recipe`` clears any stale value from a prior failure. """ try: asset = Asset.objects.get(asset_id=asset_id) @@ -535,6 +342,10 @@ def _set_processing_error(asset_id: str, message: str) -> None: return metadata = dict(asset.metadata or {}) metadata['error_message'] = message + if recipe: + metadata['error_recipe'] = recipe + else: + metadata.pop('error_recipe', None) # Disable the row alongside clearing is_processing. The viewer's # scheduling.generate_asset_list filters on ``is_enabled`` + date # window only — it doesn't check ``metadata.error_message`` — @@ -651,10 +462,18 @@ class _NormalizeAssetTask(Task): # type: ignore[type-arg] Mirrors the YouTube task's failure handling: clears ``is_processing`` and writes ``metadata.error_message`` so the operator's table row never stays stuck on "Processing" after a - crash. The message is the str() of the exception so it surfaces - something concrete (``UnidentifiedImageError: cannot identify - image file '/data/anthias_assets/abc.heic'``) without leaking a - full traceback into the API response. + crash. Two message shapes: + + * ``UnsupportedVideoCodecError`` — the gate's user-facing + exception. The message body is already operator-readable + (no class-name prefix); any attached ``recipe`` lands in + ``metadata.error_recipe`` so the modal can render it in a + copyable ```` block. + * Anything else (corrupt HEIC, ffmpeg subprocess error, ...) — + prefix with the exception class so the operator sees a + concrete signal (``UnidentifiedImageError: cannot identify + image file '/data/.../abc.heic'``) without leaking the full + traceback. """ def on_failure( @@ -669,7 +488,10 @@ def on_failure( if not asset_id: return try: - _set_processing_error(asset_id, f'{type(exc).__name__}: {exc}') + if isinstance(exc, UnsupportedVideoCodecError): + _set_processing_error(asset_id, str(exc), recipe=exc.recipe) + else: + _set_processing_error(asset_id, f'{type(exc).__name__}: {exc}') _notify(asset_id) except Exception: logging.exception( @@ -836,18 +658,10 @@ def _drop_image_staging() -> None: # --------------------------------------------------------------------------- -# Video normalisation: ffprobe → passthrough or libx264/aac transcode +# Video normalisation: ffprobe → metadata write + HW-decode codec gate # --------------------------------------------------------------------------- -# How long a single transcode attempt is allowed to run. 30 minutes -# matches the ceiling called out in the issue; a 1080p H.264-source -# transcode of a typical 5-minute clip on a Pi 5 finishes in well -# under 2 minutes. The hard ceiling is the bound for the pathological -# case (mis-routed long-form upload, hung ffmpeg). Once exceeded, -# Celery kills the worker process and on_failure clears is_processing. -NORMALIZE_VIDEO_TIME_LIMIT_S = 60 * 30 - # Wall-clock cap on the ffprobe call. A working ffprobe answers in # under a second on small files; 60s covers a stalled-IO worst case # and stops a hung process from blocking the worker indefinitely. @@ -880,11 +694,27 @@ def _ffprobe_streams(input_path: str) -> dict[str, Any]: def _ffprobe_summary(input_path: str) -> dict[str, Any]: """Reduce ffprobe's payload to the dimensions we branch on. - Returns a dict with four keys, all populated: + Returns a dict with these keys, all populated: * ``container`` — lowercase format token, ``'unknown'`` if ffprobe couldn't decide. * ``video_codec`` — lowercase codec name, ``'unknown'`` if the file has no video stream or the probe failed. + * ``video_pixels`` — ``width * height`` for the first video + stream, ``None`` if no video stream or dimensions missing. + Convenience for callers comparing against a total-pixel + budget. + * ``video_width`` / ``video_height`` — per-axis dimensions + of the first video stream, ``None`` if no video stream or + dimensions missing. The envelope passthrough check uses + these axis-by-axis (an ultrawide 5760×1080 source has fewer + total pixels than 4K but exceeds the width of a 3840×2160 + envelope and must transcode). + * ``video_fps`` — the first video stream's average frame rate + as a float, or ``None`` if no video stream or + ``r_frame_rate`` was unparseable. Used by the playback- + envelope transcode to decide whether to emit + ``-r envelope.max_fps`` (only when source > cap; the cap + is one-way and never up-converts sub-cap content). * ``audio_codec`` — lowercase codec name, ``'none'`` when the file genuinely carries no audio stream, or ``'unknown'`` if the audio stream existed but ffprobe couldn't name its @@ -911,6 +741,10 @@ def _ffprobe_summary(input_path: str) -> dict[str, Any]: return { 'container': 'unknown', 'video_codec': 'unknown', + 'video_pixels': None, + 'video_width': None, + 'video_height': None, + 'video_fps': None, 'audio_codec': 'unknown', 'duration_seconds': None, } @@ -924,29 +758,47 @@ def _ffprobe_summary(input_path: str) -> dict[str, Any]: None, ) fmt_data = probe.get('format') or {} - # Container resolution prefers ffprobe's ``format.format_name`` - # over the filename extension: a ``.mov`` file that's actually - # MKV bytes (or a ``.bin`` extension hiding an mp4) would be - # mis-classified as passthrough-eligible if we trusted the - # filename. ffprobe reports a comma-joined list of synonyms - # (e.g. ``mov,mp4,m4a,3gp,3g2,mj2``) — we accept the - # passthrough decision if ANY token matches the supported set. - # Falls back to the extension only when ffprobe couldn't - # populate format_name (probe failed; shouldn't happen here - # since we already returned 'unknown' above on probe error). + # ffprobe reports a comma-joined synonym list in + # ``format.format_name`` (e.g. ``mov,mp4,m4a,3gp,3g2,mj2`` for + # the QuickTime family). The first token is the canonical + # container name; we keep it as-is for the metadata surface so + # the operator UI can show "mp4" for an mp4-family upload + # without having to know about the synonyms. fmt = fmt_data.get('format_name') or '' fmt_tokens = [t.strip().lower() for t in fmt.split(',') if t.strip()] if fmt_tokens: - # Pick the first token that's in the passthrough set; if - # none match, take the first reported token verbatim so - # downstream branching produces a deterministic 'unknown'. - container = next( - (t for t in fmt_tokens if t in _PASSTHROUGH_CONTAINERS), - fmt_tokens[0], - ) + container = fmt_tokens[0] else: container = _ext(input_path).lstrip('.') or 'unknown' video_codec = ((video or {}).get('codec_name') or 'unknown').lower() + # Width × height for the metadata surface. ffprobe returns + # ``width`` / ``height`` only for video streams, and only when + # the demuxer could decide. A missing or unparseable value + # collapses to None. + try: + vw = int((video or {}).get('width') or 0) + vh = int((video or {}).get('height') or 0) + except (TypeError, ValueError): + vw = vh = 0 + video_width: int | None = vw if vw > 0 else None + video_height: int | None = vh if vh > 0 else None + video_pixels: int | None = vw * vh if vw > 0 and vh > 0 else None + # Average frame rate, used by the envelope transcode to decide + # whether to emit ``-r``. ffprobe writes ``r_frame_rate`` as a + # rational ``num/den`` string (e.g. ``30000/1001`` for NTSC, + # ``60/1`` for true 60 fps). Anything unparseable collapses to + # ``None`` so the caller treats it as "we can't tell" and skips + # the fps gate (the codec / resolution gates still fire). + video_fps: float | None = None + raw_fps = (video or {}).get('r_frame_rate') + if raw_fps and isinstance(raw_fps, str) and '/' in raw_fps: + num_str, _, den_str = raw_fps.partition('/') + try: + num, den = float(num_str), float(den_str) + if den > 0: + video_fps = num / den + except ValueError: + video_fps = None if audio is None: audio_codec = 'none' else: @@ -969,260 +821,216 @@ def _ffprobe_summary(input_path: str) -> dict[str, Any]: return { 'container': container, 'video_codec': video_codec, + 'video_pixels': video_pixels, + 'video_width': video_width, + 'video_height': video_height, + 'video_fps': video_fps, 'audio_codec': audio_codec, 'duration_seconds': duration_seconds, } -def _video_can_passthrough( - summary: dict[str, Any], - profile: _BoardProfile | None = None, -) -> bool: - """``True`` if the file is in a format the *target board's* viewer - plays directly. - - The probe needs to answer "yes" to all three questions: is the - container one we accept; is the video codec one the board's - player handles (H.264 only on pi2/pi3 — they have no HEVC - hardware and an A53 CPU can't software-decode HEVC at 1080p; H.264 - + HEVC on pi4-64/pi5/x86); is the audio codec one of the - demuxer-compatible set (or absent). Any 'unknown' answer (probe - failed, exotic codec) triggers a transcode — better to spend the - cycles than to let an unplayable file sit in the rotation. - - ``profile`` defaults to the board profile resolved from - ``DEVICE_TYPE`` so callers don't have to thread it through. Tests - pass a specific profile to assert per-board behaviour without - mutating the env. +_VIDEO_METADATA_KEYS = ( + 'container', + 'video_codec', + 'video_width', + 'video_height', + 'video_fps', + 'audio_codec', +) + + +# Per-board hardware-decode codec set. Mirrors +# ``anthias_viewer.media_player._PI_HWDEC_BY_CODEC`` — the two tables +# must list the same codecs for each board, because the upload-side +# gate decides what to accept and the playback-side dispatch decides +# how to play. If they drift, an asset accepted at upload will fail +# at the viewer with mpv's silent SW-decode fallback (which the gate +# exists to prevent). +# +# Empty / missing entry means "no codec on this device decodes in +# hardware" — every video upload is rejected. The catch-all ``arm64`` +# DEVICE_TYPE lands here when ``anthias_host_agent`` hasn't published +# a more specific subtype to Redis; an unknown aarch64 SBC isn't +# guaranteed to have a v4l2_request decoder mpv can address, so we +# refuse rather than ship a clip that would SW-decode at play time. +_HW_DECODE_VIDEO_CODECS: dict[str, frozenset[str]] = { + 'pi2': frozenset({'h264'}), + 'pi3': frozenset({'h264'}), + 'pi4-64': frozenset({'h264', 'hevc'}), + 'pi5': frozenset({'hevc'}), + 'rockpi4': frozenset({'h264', 'hevc'}), + 'x86': frozenset({'h264', 'hevc'}), +} + + +def _hw_decoded_codecs() -> frozenset[str]: + """Codecs the *current* board can hardware-decode through mpv. + + Resolves ``DEVICE_TYPE`` via ``anthias_common.board.resolve_device_key`` + so a Rock Pi 4 running the catch-all ``arm64`` image still picks + up its ``{h264, hevc}`` set once ``anthias_host_agent`` publishes + ``host:board_subtype=rockpi4``. An unknown / unrecognised + DEVICE_TYPE returns the empty set so every video gets rejected. """ - if profile is None: - profile = _resolve_board_profile() - if summary.get('container') not in _PASSTHROUGH_CONTAINERS: - return False - if summary.get('video_codec') not in profile['passthrough_video_codecs']: - return False - if summary.get('audio_codec') not in _PASSTHROUGH_AUDIO_CODECS: - return False - return True - - -def _transcode_to_target( - input_path: str, - output_path: str, - profile: _BoardProfile | None = None, -) -> None: - """Run a libx264 or libx265 transcode picked by the board profile. - - Profile decides codec + encoder args; the *invariants* are: - - * ``-y`` and ``-nostdin`` keep ffmpeg non-interactive (it would - otherwise prompt on overwrite or block waiting for input). - * ``-threads 2`` caps CPU usage so the viewer keeps two cores - free on Pi 4 / Pi 5; combined with the ``nice -n 19 ionice -c - 3`` wrapper on the celery worker this means a transcode - effectively never disrupts active playback. libx265 honours the - same flag and parallelises within those two threads. - * ``-c:a aac -b:a 192k`` matches every Anthias-supplied default - asset's audio profile, regardless of video codec. - * ``-movflags +faststart`` shifts the moov atom to the front of - the file so playback can begin before the file is fully - buffered — relevant when the viewer is fed via an HTTP serve - later, and harmless otherwise. - - The ``profile`` parameter lets callers (read: tests) override the - env-resolved profile so a single host can exercise both the - libx264 and libx265 branches without mutating ``DEVICE_TYPE``. + return _HW_DECODE_VIDEO_CODECS.get(resolve_device_key(), frozenset()) + + +def _ffmpeg_reencode_recipe( + supported: frozenset[str], + source_filename: str = '', +) -> str: + """Return an ``ffmpeg`` command line the operator can run on + their workstation to transcode an unsupported upload into a + codec this board hardware-decodes. + + Prefers libx264 when H.264 is in the board's supported set — + libx264 is roughly 5-10× faster than libx265 at comparable + quality, which matters when the operator is doing the encode by + hand. Falls back to libx265 + ``-tag:v hvc1`` for Pi 5 (HEVC- + only board). Returns an empty string when the board has no HW + decode set at all — there's nothing the operator can transcode + to that would land in a supported pipe. + + ``source_filename``, when supplied, substitutes the bare upload + filename (no path) for the ``INPUT`` placeholder and reuses its + stem for ``OUTPUT.mp4`` so the operator can copy the recipe + verbatim into their terminal without hand-editing it. """ - if profile is None: - profile = _resolve_board_profile() - sh.ffmpeg( - '-y', - '-nostdin', - '-threads', - '2', - '-i', - input_path, - *profile['video_args'], - *_AUDIO_TRANSCODE_ARGS, - '-movflags', - '+faststart', - output_path, - _timeout=NORMALIZE_VIDEO_TIME_LIMIT_S, - ) + if 'h264' in supported: + template = ( + 'ffmpeg -i {input} -c:v libx264 -preset medium -crf 23 ' + '-c:a aac -b:a 192k -movflags +faststart {output}' + ) + target_suffix = 'h264' + elif 'hevc' in supported: + template = ( + 'ffmpeg -i {input} -c:v libx265 -preset medium -crf 28 ' + '-tag:v hvc1 -c:a aac -b:a 192k -movflags +faststart ' + '{output}' + ) + target_suffix = 'hevc' + else: + return '' + if source_filename: + # ``shlex.quote`` produces a safe shell-quoted form for any + # filename — single quotes, spaces, ``;``, ``$()``, etc. all + # land inside literal quoting that the operator can paste + # without re-editing. The output filename carries the + # target-codec suffix (``sample.h264.mp4`` / ``sample.hevc.mp4``) + # so a recipe whose input shares the output stem doesn't ask + # the operator to overwrite their source. + in_quoted = shlex.quote(source_filename) + stem, _ = path.splitext(source_filename) + out_quoted = shlex.quote(f'{stem}.{target_suffix}.mp4') + else: + in_quoted = 'INPUT' + out_quoted = f'OUTPUT.{target_suffix}.mp4' + return template.format(input=in_quoted, output=out_quoted) + +class UnsupportedVideoCodecError(Exception): + """Raised by ``_run_video_normalisation`` when a video upload's + codec can't be hardware-decoded on this device. -def _resolve_duration_seconds(uri: str) -> int | None: - """ffprobe-driven duration for the post-transcode row. - - Used only on the transcode branch (where the file path changed - so the summary's pre-transcode duration is no longer - representative). The passthrough branch reuses the duration - pulled from ``_ffprobe_summary`` to avoid a second ffprobe shell. - - Returns ``None`` when: - * ffprobe is unavailable in this environment - (``get_video_duration`` returns None on CommandNotFound), - * the probe ran but couldn't extract a duration line, OR - * the probe raised any exception. - - The exception-swallowing branch matters: ``get_video_duration`` - raises on ``sh.ErrorReturnCode_1`` ("Bad video format") and on - bare ``Exception`` for unexpected failures. After a successful - transcode the file is on disk and the row is otherwise ready — - failing the *whole task* because the post-transcode duration - probe stumbled would be an own-goal. Keep duration best-effort - and let the operator edit the row's duration manually if - needed. + Carries the suggested ``recipe`` (an ``ffmpeg`` command the + operator can run to fix the upload) as an attribute so + ``_NormalizeAssetTask.on_failure`` can persist it alongside the + human-readable message — the UI surfaces the two in different + spots (message inline, recipe in a copyable ```` block). """ - try: - delta = get_video_duration(uri) - except Exception: - logging.exception( - 'normalize_video_asset: post-transcode duration probe ' - 'failed for %s; leaving duration unset', - uri, - ) - return None - if delta is None: - return None - return max(1, int(delta.total_seconds())) + + def __init__(self, message: str, recipe: str = '') -> None: + super().__init__(message) + self.recipe = recipe def _run_video_normalisation(asset: Asset) -> None: + """Probe the upload, record what ffprobe finds in ``metadata``, + and reject the asset if its codec isn't hardware-decoded on this + device. + + The file is never rewritten. Anthias does not re-encode video + on-device — every modern board the viewer supports already + hardware-decodes its accepted codec set (H.264 + HEVC on most + boards; HEVC only on Pi 5; H.264 only on Pi 2 / Pi 3), and the + on-device libx265 / libx264 transcode path tried in earlier + revisions wedged a Pi 4's celery worker for 99 minutes on a + single 4K60 H.264 → HEVC pass before zombieing. + + Uploading a codec outside the board's HW set is rejected — the + viewer would otherwise fall through to mpv's software decode and + show drops the operator paid for hardware to avoid. The metadata + fields written before the rejection let the operator see what + they uploaded (codec / dims / fps) alongside the error message. + """ asset_id = asset.asset_id src_uri = asset.uri or '' if not src_uri or not path.isfile(src_uri): raise FileNotFoundError(f'video source missing: {src_uri!r}') - src_ext = _ext(src_uri) summary = _ffprobe_summary(src_uri) - profile = _resolve_board_profile() metadata = dict(asset.metadata or {}) - metadata['original_ext'] = src_ext metadata.pop('error_message', None) + for key in _VIDEO_METADATA_KEYS: + value = summary.get(key) + if value is not None: + metadata[key] = value - if _video_can_passthrough(summary, profile): - # No re-encode. Keep the file at its current uri; flip the - # in-progress flag and write the duration if ffprobe could - # answer for it. Recording ``transcode_target`` even on the - # passthrough path keeps an operator's metadata view - # consistent — they can see "this device wanted hevc, the - # upload already was hevc, no work needed" without inferring. - metadata['transcoded'] = False - metadata['transcode_target'] = profile['transcode_target'] - update: dict[str, Any] = { - 'is_processing': False, - 'metadata': metadata, - } - # Reuse the duration from ``_ffprobe_summary`` rather than - # re-shelling ffprobe via ``get_video_duration``: the file - # didn't move, so the summary's value is authoritative. - # Saves one ffprobe invocation per passthrough row — the - # common case on a per-board-codec-matched fleet. - passthrough_duration = summary.get('duration_seconds') - if isinstance(passthrough_duration, int): - update['duration'] = passthrough_duration - Asset.objects.filter(asset_id=asset_id).update(**update) - _notify(asset_id) - return - - # Transcode. Output lives next to the source as ``.mp4``. - # The staging file uses a `.staging.mp4` suffix rather than - # ``.mp4.tmp`` because ffmpeg picks the muxer from the output - # extension; ``.tmp`` makes it bail with "Unable to choose an - # output format". The staging-file suffix sits inside the same - # mtime guard as cleanup() so a crash mid-transcode still gets - # GCed by the orphan-file sweep. - base_no_ext = path.splitext(src_uri)[0] - final_uri = f'{base_no_ext}.mp4' - # ``staging`` deliberately uses a ``.staging.mp4`` suffix rather - # than ``.mp4.tmp``: ffmpeg picks its muxer from the output - # extension, and ``.tmp`` makes it bail with "Unable to choose an - # output format". The suffix also guarantees ``staging != src_uri`` - # for the in-place transcode case (a non-h264 ``.mp4`` whose - # ``base_no_ext`` matches): ffmpeg keeps reading from src_uri while - # writing to a distinct path. ``os.replace`` then atomically swaps - # the input out for the transcoded output. - staging = f'{base_no_ext}.staging.mp4' - - def _drop_staging() -> None: - # All transcode failure paths converge through this helper so - # a partially-written staging file never lingers after a raise. - # cleanup() would eventually GC it as an orphan, but doing it - # inline keeps /anthias_assets/ free of debris an operator - # might trip over. - try: - os.remove(staging) - except OSError: - pass - - try: - _transcode_to_target(src_uri, staging, profile) - except sh.TimeoutException as exc: - # Time-limit overruns are surfaced as TimeoutException; let - # on_failure land so is_processing clears. - _drop_staging() - raise RuntimeError(f'ffmpeg timed out for {src_uri!r}: {exc}') from exc - except sh.ErrorReturnCode as exc: - _drop_staging() - # ``exc.stderr`` is bytes; ``!r`` would render it as - # ``b'...'`` in the operator-facing metadata.error_message. - # Decode + trim the tail for readability — ffmpeg's last few - # lines of stderr are usually the diagnostic, the rest is - # build-info noise. - raise RuntimeError( - f'ffmpeg failed for {src_uri!r}: {_format_subprocess_stderr(exc)}' - ) from exc - - if not path.isfile(staging) or os.stat(staging).st_size == 0: - # ffmpeg sometimes returns exit 0 but produces an empty file - # (broken stream, silent codec mismatch). Reject the result - # and clean up the empty file rather than promoting it. - _drop_staging() - raise RuntimeError(f'ffmpeg produced no output for {src_uri!r}') - - # Same rename-failure cleanup as the image pipeline: the atomic - # rename normally succeeds in <1ms, but a filesystem-full / - # permissions / cross-device error here would otherwise leave - # the staging file hanging around. Mirror the contract by - # dropping it on any OSError. - try: - os.replace(staging, final_uri) - except OSError: - _drop_staging() - raise - - # Drop the original if it lived under a different name (e.g. a - # ProRes .mov whose transcoded H.264 lands at the same base.mp4). - if final_uri != src_uri: - try: - os.remove(src_uri) - except OSError: - logging.exception( - 'normalize_video_asset: removing original %s failed', - src_uri, - ) - - duration = _resolve_duration_seconds(final_uri) - - metadata['transcoded'] = True - # ``transcode_target`` records what we *aimed* to produce so an - # operator can see "this row was re-encoded to hevc on a pi5 - # device" without re-probing the file. The actual codec landed in - # the file is identical to this target — ffmpeg only deviates - # silently if the encoder is unavailable, which is fatal at this - # point (libx265 ships in the apt ffmpeg build for every Anthias - # board, see the configure flags in image_builder). - metadata['transcode_target'] = profile['transcode_target'] update_dict: dict[str, Any] = { - 'uri': final_uri, 'mimetype': 'video', - 'is_processing': False, 'metadata': metadata, } - if duration is not None: - update_dict['duration'] = duration - Asset.objects.filter(asset_id=asset_id).update(**update_dict) + duration_seconds = summary.get('duration_seconds') + if isinstance(duration_seconds, int) and duration_seconds > 0: + update_dict['duration'] = duration_seconds + + src_codec = (summary.get('video_codec') or '').lower() + supported = _hw_decoded_codecs() + if src_codec in supported: + update_dict['is_processing'] = False + Asset.objects.filter(asset_id=asset_id).update(**update_dict) + _notify(asset_id) + return - _notify(asset_id) + # Codec is outside the board's HW decode set (or ffprobe couldn't + # read it). Commit the metadata we *did* gather so the operator's + # asset-list row carries the rejected codec / dims / fps, then + # raise so ``_NormalizeAssetTask.on_failure`` fills in + # ``error_message`` and clears ``is_processing``. + Asset.objects.filter(asset_id=asset_id).update(**update_dict) + display_codec = ( + src_codec if src_codec and src_codec != 'unknown' else 'unknown' + ) + # ``upload_name`` is stashed by the dashboard / API at upload + # time — the on-disk file gets renamed to ``.`` but + # the recipe wants a name the operator can paste straight into + # their workstation terminal. YouTube / pre-rebrand rows that + # don't carry the field fall through to a stable + # ``upload`` placeholder so the recipe still teaches the + # operator the input extension. + upload_name = metadata.get('upload_name') or ( + f'upload{path.splitext(src_uri)[1]}' + ) + recipe = _ffmpeg_reencode_recipe(supported, upload_name) + if supported: + supported_str = ', '.join(sorted(supported)) + message = ( + f'Video codec {display_codec!r} is not hardware-decoded on ' + f'this device. Supported: {supported_str}.' + ) + else: + # Empty ``supported`` means we hit the catch-all ``arm64`` + # branch — DEVICE_TYPE is set but host_agent never published + # ``host:board_subtype`` so we can't certify any codec. Say + # so rather than the misleading "Supported: none." which + # reads like the board has no decoder at all. + message = ( + f'Video codec {display_codec!r} can not be verified for ' + 'hardware decoding on this device — the board has not ' + 'reported a known subtype. Re-flash with the board-' + 'specific image (e.g. Rock Pi 4) so anthias_host_agent ' + 'can publish its capabilities.' + ) + raise UnsupportedVideoCodecError(message, recipe=recipe) diff --git a/src/anthias_viewer/media_player.py b/src/anthias_viewer/media_player.py index 679f793a3..c82002277 100644 --- a/src/anthias_viewer/media_player.py +++ b/src/anthias_viewer/media_player.py @@ -1,8 +1,9 @@ import logging import os import subprocess -from typing import ClassVar +from typing import IO, ClassVar +from anthias_common.board import ARM64_DEVICE_TYPES, resolve_device_key from anthias_common.device_helper import get_device_type from anthias_common.utils import clamp_screen_rotation from anthias_server.settings import settings @@ -163,7 +164,7 @@ def get_alsa_audio_device() -> str: # ALSA card names below (vc4hdmi*, "Headphones") don't exist on # those boards, so route via DEVICE_TYPE env first and only fall # through to the Pi-name dispatch when we're actually on a Pi. - if os.environ.get('DEVICE_TYPE') == 'arm64': + if os.environ.get('DEVICE_TYPE') in ARM64_DEVICE_TYPES: # No portable per-SoC HDMI card name across Rockchip / # Allwinner / Amlogic, so defer to ALSA's `default` device. # Operators with a non-standard HDMI sink can override via @@ -207,6 +208,112 @@ def is_playing(self) -> bool: raise NotImplementedError +# Per-codec hwdec preference on Pi, per-board: +# +# Pi 4: H.264 → v4l2m2m-copy (V3D V4L2 M2M decoder via +# bcm2835-codec, up to 1080p60) +# HEVC → drm-copy (FFmpeg v4l2_request_hevc, up to +# 4Kp60 via the dedicated HEVC block) +# +# Pi 5: H.264 → auto-copy (Hantro G1 silicon exists but mpv +# has no v4l2-request H.264 hwdec +# upstream; passing v4l2m2m-copy +# here would just log "Could not +# find a valid device" errors before +# silently SW-falling-back. The +# playback envelope (HEVC 4Kp60) means +# every asset normalised post-rollout +# lands as HEVC, so this branch only +# fires for legacy variants the +# re-render walker hasn't caught yet.) +# HEVC → drm-copy (Hantro G2, up to 4Kp60. Requires +# `dtoverlay=vc4-kms-v3d,cma-512` in +# /boot/firmware/config.txt — the +# stock 64 MB CMA region can't fit +# a 4K HEVC dst buffer pool, and the +# kernel cmdline `cma=` route silently +# orphans the rpi-hevc-dec driver.) +# +# generic-arm64 (Armbian SBCs — Rock Pi 4, Orange Pi, etc.): +# H.264 → v4l2m2m-copy (RK3399 Hantro via the v4l2m2m +# HEVC → v4l2m2m-copy driver. mpv's --hwdec=help on the +# latest-generic-arm64 image lists +# both h264_v4l2m2m and hevc_v4l2m2m; +# on boards without a working driver +# mpv logs a warning and SW-falls- +# back at runtime.) +# +# `auto-copy` is the universal safe fallback when ffprobe can't +# read the codec (missing file, network URI we don't probe, etc.). +# +# An earlier revision did this with a Lua on_load hook, but +# video-codec-name is empty at every event mpv exposes to scripts +# before hwdec init (on_load, on_preloaded). ffprobing from Python +# at launch time is both simpler and the only thing that actually +# works. +_PI_HWDEC_BY_CODEC: dict[str, dict[str, str]] = { + 'pi4-64': {'h264': 'v4l2m2m-copy', 'hevc': 'drm-copy'}, + 'pi5': {'hevc': 'drm-copy'}, + # Rock Pi 4 (RK3399, Radxa). Both codecs go through + # ``drm-copy``: the arm64 viewer image now pulls the Raspberry + # Pi repo's ffmpeg (``+rpt1``, with ``--enable-v4l2-request`` + # — same package family as Pi 4 / Pi 5), so mpv exposes the + # stateless v4l2_request decoders that the RK3399's ``rkvdec`` + # (HEVC) and ``rockchip,rk3399-vpu-dec`` Hantro VPU (H.264) + # implement. The ``start_viewer.sh`` entrypoint creates the + # ``/dev/video-dec*`` symlinks the v4l2_request decoder + # discovery code expects (privileged docker mounts its own + # /dev tmpfs without udev's symlinks). + 'rockpi4': {'h264': 'drm-copy', 'hevc': 'drm-copy'}, +} + + +def _probe_video_codec(uri: str) -> str: + """Return the canonical lowercase video codec name for ``uri``. + + Empty string on probe failure (missing file, unreadable codec, + ffprobe absent, etc.) — callers should then pick a safe + fallback like ``auto-copy``. Short timeout because this runs + synchronously before every mpv launch. + """ + try: + result = subprocess.run( + [ + 'ffprobe', + '-v', + 'error', + '-select_streams', + 'v:0', + '-show_entries', + 'stream=codec_name', + '-of', + 'default=nw=1:nk=1', + uri, + ], + capture_output=True, + text=True, + timeout=5, + ) + return result.stdout.strip().lower() + except (subprocess.SubprocessError, OSError): + return '' + + +def _pi_hwdec_for_uri(uri: str) -> str: + """mpv ``--hwdec=`` value for ``uri`` on Pi 4 / Pi 5 / Rock Pi 4. + + Reads the board key via ``resolve_device_key``, which upgrades a + catch-all ``arm64`` DEVICE_TYPE to the specific subtype the + host_agent publishes (Rock Pi 4 → ``rockpi4``). ``auto-copy`` is + the safe fallback when ffprobe can't read the codec; the upload + gate (`anthias_server.processing._hw_decoded_codecs`) prevents + any codec outside ``_PI_HWDEC_BY_CODEC`` reaching this branch in + the first place. + """ + board_map = _PI_HWDEC_BY_CODEC.get(resolve_device_key(), {}) + return board_map.get(_probe_video_codec(uri), 'auto-copy') + + class MPVMediaPlayer(MediaPlayer): def __init__(self) -> None: MediaPlayer.__init__(self) @@ -225,61 +332,142 @@ def play(self) -> None: # reads the connector's EDID-preferred mode (4K on most modern # TVs) and runs CPU zimg upscale, which drops below real-time # on the A72. Software decode of 1080p H.264 fits 4 cores fine. + # Pi 5 keeps the same tuning on the cage path — it doesn't + # hurt, and mpv ignores --drm-mode under --vo=gpu + # --gpu-context=wayland anyway. device_type = os.environ.get('DEVICE_TYPE', '') extra_args: list[str] = [] - if device_type in ('pi4-64', 'pi5'): + if device_type == 'pi4-64': extra_args = [ '--drm-mode=1920x1080@60', '--vd-lavc-threads=4', ] - - # x86 runs under `cage` (a wlroots kiosk compositor — see - # bin/start_viewer.sh); cage holds DRM master, so --vo=drm is - # denied. Route mpv through the GL VO over a Wayland EGL - # context, which is the generic path mpv supports on every x86 - # GPU with Mesa or vendor GL drivers. Paired with - # --hwdec=auto-safe, VAAPI-capable iGPUs (Intel iHD/i965, AMD - # radeonsi, …) decode in hardware and hand frames to the GL - # context as DMA-BUFs via dmabuf-interop-gl; software decode - # still works via the same VO for codecs without HW support. - # --vo=dmabuf-wayland would skip the GL upload entirely but - # segfaults under cage in the viewer's background-spawn path - # (mpv 0.40.0 + wlroots-0.18 + libplacebo dies between hwdec - # init and file open). Pi boards (pi4-64/pi5) keep --vo=drm — - # they own the framebuffer directly with no compositor. - # arm64 (non-Pi ARM SBCs on Armbian) goes the same - # route as x86 because the viewer container is wrapped in - # cage there too; hwdec is best-effort per SoC. - if device_type in ('x86', 'arm64'): + elif device_type == 'pi5': + extra_args = ['--vd-lavc-threads=4'] + + # Per-board VO selection: + # + # * x86 / arm64 / pi5 run under `cage` (a wlroots kiosk + # compositor — see bin/start_viewer.sh); cage holds DRM + # master, so --vo=drm is denied. mpv goes through the GL + # VO over a Wayland EGL context. Paired with + # --hwdec=auto-safe, VAAPI-capable iGPUs on x86 (Intel + # iHD/i965, AMD radeonsi, …) decode in hardware and hand + # frames to the GL context as DMA-BUFs via + # dmabuf-interop-gl; software decode still works via the + # same VO. Pi 5's V3D 7.1 has enough bandwidth to composite + # at the connector's native mode (typically 4K) on top of + # software-decoded video. arm64 is best-effort per SoC. + # --vo=dmabuf-wayland would skip the GL upload entirely + # but segfaults under cage (mpv 0.40 + wlroots-0.18 + + # libplacebo dies between hwdec init and file open). + # + # * Pi4-64 stays on Qt linuxfb (no compositor) with mpv's + # --vo=drm. The V3D 6.0 doesn't have the bandwidth to + # composite cage on top of software-decoded video at 4K + # (738 vo drops/30 s in testing). mpv's --vo=drm does its + # own DRM master juggling — briefly grabbing master, + # rendering, dropping back — which coexists with Qt + # linuxfb in a way that --vo=gpu --gpu-context=drm does + # not (Mesa GBM holds master persistently and contends + # with Qt's framebuffer use, manifesting as "Failed to + # acquire DRM master: Permission denied"). So Pi 4 stays + # on --vo=drm + --drm-mode=1920x1080@60 — the production + # path inherited from master. + # ``generic-arm64`` is the legacy label that pre-rename arm64 + # images carry — it shares the cage + Wayland stack with + # ``arm64`` and must take the same VO path. Without this the + # Rock Pi 4 (still on a generic-arm64 image) falls into the + # ``--vo=drm`` else branch and mpv aborts with "No primary + # DRM device could be picked" because cage holds DRM master. + if device_type in ('x86', 'arm64', 'generic-arm64', 'pi5'): vo_args = ['--vo=gpu', '--gpu-context=wayland'] else: vo_args = ['--vo=drm'] - # Issue #2856. On x86, cage/wlroots is rotated via wlr-randr and - # mpv's wayland VO inherits the compositor transform — passing - # --video-rotate here too would double-rotate. On Pi --vo=drm - # writes straight to the framebuffer with no compositor in the - # path, so the rotation has to happen inside mpv. Skip the arg - # at 0° so unrotated displays stay on the existing CLI. + # Rotation: cage/wlroots is rotated via wlr-randr (issue + # #2856, the wiring lives in src/anthias_viewer/__init__.py) + # and mpv's wayland VO inherits the compositor transform + # automatically — passing --video-rotate would double-rotate + # there. On Pi 4 (linuxfb, no compositor) mpv has to apply + # the transform itself. rotation = _screen_rotation() rotate_args: list[str] = [] - if rotation and device_type != 'x86': + if rotation and device_type == 'pi4-64': rotate_args = [f'--video-rotate={rotation}'] + # Hwdec selection. Strategy summary: + # + # * x86 / arm64 → `--hwdec=auto-copy`. Picks vaapi-copy on + # Intel/AMD iGPUs (the only HW decode method mpv 0.40 ships + # with for x86 outside of NVIDIA-specific options); on + # arm64 there's nothing in auto-copy that matches + # Rockchip's V4L2 stateless decoder, so it falls back to + # software. arm64 HW decode via a vendor-tuned plugin is a + # Tier-2 follow-up. + # * Pi 4 / Pi 5 → ffprobe the asset (~50 ms for a local + # file) and pick per-codec, because `auto-copy`'s upstream + # whitelist deliberately excludes v4l2m2m-copy (H.264 V3D + # M2M is the path Pi 4 needs). See _pi_hwdec_for_uri(). + # ffprobe-and-dispatch on any board the per-codec map covers + # (Pi 4 / Pi 5 / Rock Pi 4 via host_agent subtype). x86, the + # arm64 catch-all without a subtype, and Pi 2 / Pi 3 fall + # through to ``auto-copy`` and don't pay the ffprobe cost. + if resolve_device_key() in _PI_HWDEC_BY_CODEC: + hwdec_value = _pi_hwdec_for_uri(self.uri) + else: + hwdec_value = 'auto-copy' + + # ANTHIAS_DEBUG_DROPS=1: when set on the viewer container, + # mpv's stdout/stderr go to a host-bound log instead of + # /dev/null, *and* --no-terminal is dropped so mpv's normal + # status line ("AV: 00:00:30 / ... Dropped: N") is emitted. + # The log records hwdec-current / VO init banners plus + # per-file drop counts so reviewers can validate the test + # bed without rebuilding the image. Default (unset) + # preserves the silent stdout/stderr=/dev/null behaviour. + debug_drops = os.environ.get('ANTHIAS_DEBUG_DROPS') == '1' + terminal_args = [] if debug_drops else ['--no-terminal'] + # Popen accepts either an int sentinel (DEVNULL / STDOUT) or + # an already-opened binary IO stream for stdout/stderr. The + # int | IO[bytes] union covers both. + popen_stdout: int | IO[bytes] + popen_stderr: int | IO[bytes] + if debug_drops: + log_fd = open('/data/.anthias/mpv.log', 'ab', buffering=0) + log_fd.write(f'\n--- mpv launch {self.uri} ---\n'.encode()) + popen_stdout = log_fd + popen_stderr = subprocess.STDOUT + else: + popen_stdout = subprocess.DEVNULL + popen_stderr = subprocess.DEVNULL + + # ``--video-sync=display-resample`` overrides mpv 0.40's + # default (``audio``) which syncs video to the audio clock + # and drops VO frames when the two clocks drift. On every + # board we tested (Pi 4, Pi 5, x86) the audio-clock default + # produced 60–90% VO drops at 60 fps content even when the + # decoder was healthy (mpv reports drops at the VO, not the + # decoder). Digital signage cares about smooth video more + # than sub-frame A/V sync; display-resample syncs video to + # the display's refresh and resamples audio to match. Audio + # resampling is cheap (a 2-channel resample takes <1% CPU) + # and most signage clips have no audible content anyway. self.process = subprocess.Popen( [ 'mpv', - '--no-terminal', + *terminal_args, *vo_args, - '--hwdec=auto-safe', + f'--hwdec={hwdec_value}', + '--video-sync=display-resample', *extra_args, *rotate_args, f'--audio-device=alsa/{get_alsa_audio_device()}', '--', self.uri, ], - stdout=subprocess.DEVNULL, - stderr=subprocess.DEVNULL, + stdout=popen_stdout, + stderr=popen_stderr, ) def stop(self) -> None: @@ -359,21 +547,23 @@ class MediaPlayerProxy: @classmethod def get_instance(cls) -> MediaPlayer: if cls.INSTANCE is None: - # Force MPV (over VLC) on two device_types that otherwise + # Force MPV (over VLC) on the device_types that otherwise # match the Pi-name dispatch below: # # * pi4-64 — Qt6 + linuxfb like pi5/x86, so VLC's # GL/GLES2/XCB outputs have no parent window to draw # into. MPV renders straight to KMS via --vo=drm. - # * arm64 — device_helper.get_device_type() falls back - # to 'pi1' on any aarch64 host whose + # * arm64 / generic-arm64 — + # device_helper.get_device_type() falls back to + # 'pi1' on any aarch64 host whose # /proc/device-tree/model isn't a Pi regex match # (Rock Pi, Orange Pi, Banana Pi, …); without this # override they'd silently route to VLC, which has # no working backend on those boards (no vc4 KMS, - # no XCB under cage). + # no XCB under cage). ``generic-arm64`` covers + # pre-rename images still in the wild. device_env = os.environ.get('DEVICE_TYPE') - force_mpv = device_env in ('pi4-64', 'arm64') + force_mpv = device_env in ('pi4-64', 'arm64', 'generic-arm64') if ( get_device_type() in ['pi1', 'pi2', 'pi3', 'pi4'] and not force_mpv diff --git a/tests/test_host_agent_board_subtype.py b/tests/test_host_agent_board_subtype.py new file mode 100644 index 000000000..b6abfc011 --- /dev/null +++ b/tests/test_host_agent_board_subtype.py @@ -0,0 +1,158 @@ +"""Tests for ``anthias_host_agent``'s board subtype publisher. + +``host_agent`` runs on the host (outside any container) and writes +the resolved board subtype to Redis at ``host:board_subtype``. +Server + viewer read it to upgrade the catch-all ``arm64`` +DEVICE_TYPE into a board-specific envelope / hwdec dispatch when +the silicon supports it. We pin: + +* the device-tree → subtype mapping for known boards (Rock Pi 4); +* unknown / empty / missing device-tree all collapse to ``None``; +* the Redis publish writes the resolved subtype (or empty string) + exactly once per host_agent start. +""" + +from __future__ import annotations + +from typing import Any +from unittest import mock + +import pytest + +from anthias_host_agent.__main__ import detect_board_subtype, set_board_subtype + + +@pytest.mark.parametrize( + ('model_bytes', 'expected'), + [ + # Canonical Radxa string + null terminator (what the kernel + # actually writes — null-terminated UTF-8). + (b'Radxa ROCK Pi 4B\x00', 'rockpi4'), + # 4A / 4C variants — same RK3399 silicon, same dispatch. + (b'Radxa ROCK Pi 4A\x00', 'rockpi4'), + (b'Radxa ROCK Pi 4C\x00', 'rockpi4'), + # Whitespace + mixed case — the function strips + lowercases + # so a vendor that writes the model differently doesn't slip + # through. + (b' RADXA Rock Pi 4B\n \x00', 'rockpi4'), + # Boards we haven't profiled yet stay unknown — caller falls + # back to the conservative arm64 envelope. + (b'OrangePi 3 LTS\x00', None), + (b'Banana Pi M5\x00', None), + # Empty / null-only / whitespace-only return None. + (b'\x00', None), + (b' \n\x00', None), + (b'', None), + ], +) +def test_detect_board_subtype( + model_bytes: bytes, expected: str | None +) -> None: + """The static device-tree → subtype table is the source of + truth for board detection. Any drift between this table and + ``compute_envelope``'s expected matrix keys would silently + misroute the asset processor; the parametrise pins every cell. + """ + mocked_open = mock.mock_open(read_data=model_bytes) + with mock.patch( + 'anthias_host_agent.__main__.open', mocked_open, create=True + ): + assert detect_board_subtype() == expected + + +def test_detect_board_subtype_no_devicetree() -> None: + """A host without ``/proc/device-tree/model`` (dev container, + a non-DT bootloader, balena's restricted /proc) collapses + cleanly to ``None`` instead of raising. ``host_agent`` is + started by systemd so an uncaught exception here would loop + the unit and starve every other host-side feature.""" + with mock.patch( + 'anthias_host_agent.__main__.open', + side_effect=FileNotFoundError(), + create=True, + ): + assert detect_board_subtype() is None + + +def test_set_board_subtype_writes_resolved_value() -> None: + """When a known SBC is detected, the resolved key is the + payload written to ``host:board_subtype``.""" + fake_redis = mock.MagicMock() + with mock.patch( + 'anthias_host_agent.__main__.detect_board_subtype', + return_value='rockpi4', + ): + set_board_subtype(fake_redis) + fake_redis.set.assert_called_once_with('host:board_subtype', 'rockpi4') + + +def test_set_board_subtype_writes_empty_string_on_unknown() -> None: + """Distinguishes "ran but didn't recognise" (empty string) from + "never ran" (key missing). The server-side reader treats both + identically, but the empty write is useful for diagnostics + — an operator inspecting redis can see the host_agent has + actually run.""" + fake_redis = mock.MagicMock() + with mock.patch( + 'anthias_host_agent.__main__.detect_board_subtype', + return_value=None, + ): + set_board_subtype(fake_redis) + fake_redis.set.assert_called_once_with('host:board_subtype', '') + + +def test_set_board_subtype_does_not_raise_on_redis_failure() -> None: + """``host_agent``'s startup must not crash if ``rdb.set`` + fails (some transient redis issue). The function is best- + effort — the consumer-side reader handles missing keys.""" + fake_redis = mock.MagicMock() + fake_redis.set.side_effect = Exception('simulated redis hiccup') + with mock.patch( + 'anthias_host_agent.__main__.detect_board_subtype', + return_value='rockpi4', + ): + with pytest.raises(Exception, match='simulated redis hiccup'): + # Current contract: the host_agent surfaces redis + # failures (it's a systemd unit; restart-on-failure + # will retry). If we ever wrap this in try/except, + # update both this test and the function's docstring + # to reflect the new contract. + set_board_subtype(fake_redis) + + +def test_subscriber_loop_calls_set_board_subtype( + monkeypatch: pytest.MonkeyPatch, +) -> None: + """The ``subscriber_loop`` orchestration must call + ``set_board_subtype`` before flipping ``host_agent_ready`` + — otherwise a consumer that polls for ``host_agent_ready=true`` + and immediately reads ``host:board_subtype`` could observe + the stale (or empty) value.""" + from anthias_host_agent import __main__ as ha + + fake_redis = mock.MagicMock() + call_order: list[str] = [] + + def fake_set_subtype(rdb: Any) -> None: + call_order.append('subtype') + + def fake_set(key: str, value: Any) -> None: + if key == 'host_agent_ready': + call_order.append('ready') + + fake_redis.set.side_effect = fake_set + # ``pubsub.listen()`` blocks forever in production; mock it as + # an empty iterator so the test returns. + fake_redis.pubsub.return_value.listen.return_value = iter(()) + + import redis as redis_pkg + + monkeypatch.setattr(redis_pkg, 'Redis', lambda **kw: fake_redis) + monkeypatch.setattr(ha, 'set_board_subtype', fake_set_subtype) + + ha.subscriber_loop() + + assert call_order == ['subtype', 'ready'], ( + 'set_board_subtype must complete before host_agent_ready ' + 'flips, otherwise consumers race the publish' + ) diff --git a/tests/test_media_player.py b/tests/test_media_player.py index 24d51ad36..c3c966cd9 100644 --- a/tests/test_media_player.py +++ b/tests/test_media_player.py @@ -30,16 +30,29 @@ def mpv() -> Iterator[_MPVFixtures]: patch_device_type = patch( 'anthias_viewer.media_player.get_device_type', return_value='pi4' ) + # Default-mock the ffprobe-based codec probe so tests that pin + # DEVICE_TYPE to 'pi4-64'/'pi5' don't try to spawn a real ffprobe + # subprocess inside the test harness. Returning '' makes + # _pi_hwdec_for_uri fall back to 'auto-copy', which preserves the + # pre-ffprobe behaviour the legacy tests assert on. Tests that + # care about a specific dispatch (h264 → v4l2m2m-copy etc.) layer + # a sharper patch on top of this default. + patch_probe = patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='', + ) fixtures.mock_settings = patch_settings.start() fixtures.mock_settings.__getitem__.return_value = 'hdmi' patch_device_type.start() + patch_probe.start() try: yield fixtures finally: patch_settings.stop() patch_device_type.stop() + patch_probe.stop() @patch( @@ -47,11 +60,11 @@ def mpv() -> Iterator[_MPVFixtures]: return_value='sysdefault:CARD=vc4hdmi0', ) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_play_invokes_popen_with_expected_args( +def test_play_invokes_popen_with_expected_args_on_pi4_64( mock_popen: Any, _mock_detect: Any, mpv: _MPVFixtures ) -> None: mpv.player.set_asset('file:///test/video.mp4', 30) - with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4'}): + with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}): mpv.player.play() mock_popen.assert_called_once_with( @@ -59,7 +72,10 @@ def test_play_invokes_popen_with_expected_args( 'mpv', '--no-terminal', '--vo=drm', - '--hwdec=auto-safe', + '--hwdec=auto-copy', + '--video-sync=display-resample', + '--drm-mode=1920x1080@60', + '--vd-lavc-threads=4', '--audio-device=alsa/sysdefault:CARD=vc4hdmi0', '--', 'file:///test/video.mp4', @@ -73,6 +89,9 @@ def test_play_invokes_popen_with_expected_args( def test_play_pins_1080p_mode_on_pi4_64( mock_popen: Any, mpv: _MPVFixtures ) -> None: + # Pi 4 stays on --vo=drm (Qt linuxfb + mpv DRM master juggling) + # so the --drm-mode pin is still meaningful — it sidesteps the + # CPU zimg upscale to 4K that the A72 can't keep up with. mpv.player.set_asset('file:///test/video.mp4', 30) with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}): mpv.player.play() @@ -80,25 +99,129 @@ def test_play_pins_1080p_mode_on_pi4_64( args, _ = mock_popen.call_args assert '--drm-mode=1920x1080@60' in args[0] assert '--vd-lavc-threads=4' in args[0] - assert '--hwdec=auto-safe' in args[0] + + +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='h264', +) +@patch('anthias_viewer.media_player.subprocess.Popen') +def test_play_picks_v4l2m2m_for_h264_on_pi4_64( + mock_popen: Any, _mock_probe: Any, mpv: _MPVFixtures +) -> None: + # Pi 4 H.264 dispatches to v4l2m2m-copy (V3D V4L2 M2M); mpv's + # auto-copy whitelist excludes v4l2m2m-copy so we have to set it + # explicitly. + mpv.player.set_asset('file:///test/h264.mp4', 30) + with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}): + mpv.player.play() + args, _ = mock_popen.call_args + assert '--hwdec=v4l2m2m-copy' in args[0] + assert '--hwdec=auto-copy' not in args[0] + + +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='h264', +) +@patch('anthias_viewer.media_player.subprocess.Popen') +def test_play_picks_auto_copy_for_h264_on_pi5( + mock_popen: Any, _mock_probe: Any, mpv: _MPVFixtures +) -> None: + # Pi 5 has no upstream-mpv H.264 hwdec (Hantro G1 isn't exposed + # through v4l2-request in mpv 0.40). Passing v4l2m2m-copy here + # would just log "Could not find a valid device" errors before + # silently SW-falling-back, so we send auto-copy and let mpv's + # default selector pick (which finds nothing for H.264 on Pi 5 + # and falls to software cleanly). The asset processor re-encodes + # H.264 → HEVC on Pi 5 at upload time so this path is only hit + # for pre-existing assets during the rollout window. + mpv.player.set_asset('file:///test/h264.mp4', 30) + with patch.dict('os.environ', {'DEVICE_TYPE': 'pi5'}): + mpv.player.play() + args, _ = mock_popen.call_args + assert '--hwdec=auto-copy' in args[0] assert '--hwdec=v4l2m2m-copy' not in args[0] +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='hevc', +) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_play_pins_1080p_mode_on_pi5( +def test_play_picks_drm_copy_for_hevc_on_pi( + mock_popen: Any, _mock_probe: Any, mpv: _MPVFixtures +) -> None: + # HEVC on Pi 4 / Pi 5 goes through FFmpeg's v4l2_request_hevc, + # exposed in mpv as drm-copy. + for device_type in ('pi4-64', 'pi5'): + mock_popen.reset_mock() + mpv.player.set_asset('file:///test/hevc.mp4', 30) + with patch.dict('os.environ', {'DEVICE_TYPE': device_type}): + mpv.player.play() + args, _ = mock_popen.call_args + assert '--hwdec=drm-copy' in args[0], device_type + + +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='', +) +@patch('anthias_viewer.media_player.subprocess.Popen') +def test_play_falls_back_to_auto_copy_when_probe_fails_on_pi( + mock_popen: Any, _mock_probe: Any, mpv: _MPVFixtures +) -> None: + # If ffprobe can't read the codec (missing file, timeout, …) + # the Pi dispatch must fall back to auto-copy rather than + # passing a bogus --hwdec= value. + mpv.player.set_asset('file:///test/missing.mp4', 30) + with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}): + mpv.player.play() + args, _ = mock_popen.call_args + assert '--hwdec=auto-copy' in args[0] + + +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='h264', +) +@patch('anthias_viewer.media_player.subprocess.Popen') +def test_play_does_not_probe_on_non_pi( + mock_popen: Any, mock_probe: Any, mpv: _MPVFixtures +) -> None: + # ffprobe shouldn't run on x86 / arm64 — they go through + # --hwdec=auto-copy unconditionally and probing adds latency + # before every mpv launch. + for device_type in ('x86', 'arm64'): + mock_probe.reset_mock() + mock_popen.reset_mock() + mpv.player.set_asset('file:///test/video.mp4', 30) + with patch.dict('os.environ', {'DEVICE_TYPE': device_type}): + mpv.player.play() + mock_probe.assert_not_called() + args, _ = mock_popen.call_args + assert '--hwdec=auto-copy' in args[0], device_type + + +@patch('anthias_viewer.media_player.subprocess.Popen') +def test_play_tunes_decoder_threads_on_pi5( mock_popen: Any, mpv: _MPVFixtures ) -> None: + # Pi 5 keeps software-decode threading. mpv ignores --drm-mode + # under cage (no DRM master), and the connector's native mode is + # what cage runs at — V3D 7.1 has enough bandwidth headroom for + # the 4K composite + scale. mpv.player.set_asset('file:///test/video.mp4', 30) with patch.dict('os.environ', {'DEVICE_TYPE': 'pi5'}): mpv.player.play() args, _ = mock_popen.call_args - assert '--drm-mode=1920x1080@60' in args[0] assert '--vd-lavc-threads=4' in args[0] + assert '--drm-mode=1920x1080@60' not in args[0] @patch('anthias_viewer.media_player.subprocess.Popen') -def test_play_does_not_pin_mode_on_x86( +def test_play_omits_pi_tuning_on_x86( mock_popen: Any, mpv: _MPVFixtures ) -> None: mpv.player.set_asset('file:///test/video.mp4', 30) @@ -110,35 +233,40 @@ def test_play_does_not_pin_mode_on_x86( assert '--vd-lavc-threads=4' not in args[0] +@pytest.mark.parametrize('device_type', ['x86', 'arm64', 'pi5']) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_play_uses_wayland_vo_on_x86( - mock_popen: Any, mpv: _MPVFixtures +def test_play_uses_wayland_vo_under_cage( + mock_popen: Any, mpv: _MPVFixtures, device_type: str ) -> None: + # x86 / arm64 / pi5 run under cage (a wlroots kiosk compositor); + # cage holds DRM master, so --vo=drm would be denied. These + # boards must route through --vo=gpu --gpu-context=wayland. mpv.player.set_asset('file:///test/video.mp4', 30) - with patch.dict('os.environ', {'DEVICE_TYPE': 'x86'}): + with patch.dict('os.environ', {'DEVICE_TYPE': device_type}): mpv.player.play() args, _ = mock_popen.call_args assert '--vo=gpu' in args[0] assert '--gpu-context=wayland' in args[0] assert '--vo=drm' not in args[0] + assert '--gpu-context=drm' not in args[0] @patch('anthias_viewer.media_player.subprocess.Popen') -def test_play_uses_wayland_vo_on_arm64( +def test_play_uses_drm_vo_on_pi4_64( mock_popen: Any, mpv: _MPVFixtures ) -> None: - # arm64 runs under cage (same as x86), so mpv must go - # through --vo=gpu --gpu-context=wayland — cage holds DRM master - # and would deny --vo=drm. + # Pi 4 stays on Qt linuxfb (no cage); mpv uses --vo=drm because + # --vo=gpu --gpu-context=drm needs Mesa GBM to hold DRM master + # persistently, which contends with Qt linuxfb's framebuffer use + # ("Failed to acquire DRM master: Permission denied"). mpv.player.set_asset('file:///test/video.mp4', 30) - with patch.dict('os.environ', {'DEVICE_TYPE': 'arm64'}): + with patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}): mpv.player.play() args, _ = mock_popen.call_args - assert '--vo=gpu' in args[0] - assert '--gpu-context=wayland' in args[0] - assert '--vo=drm' not in args[0] + assert '--vo=drm' in args[0] + assert '--vo=gpu' not in args[0] @patch('anthias_viewer.media_player.subprocess.Popen') @@ -589,6 +717,25 @@ def test_get_instance_returns_mpv_for_arm64( assert isinstance(instance, MPVMediaPlayer) +def test_get_instance_returns_mpv_for_generic_arm64( + reset_media_proxy: None, +) -> None: + # Legacy ``generic-arm64`` DEVICE_TYPE label (pre-rename images + # still in the wild) must also force MPV — the Rock Pi 4 in + # particular reports this label and would crash on VLC's + # absent backend without the override. + MediaPlayerProxy.INSTANCE = None + with ( + patch( + 'anthias_viewer.media_player.get_device_type', + return_value='pi1', + ), + patch.dict('os.environ', {'DEVICE_TYPE': 'generic-arm64'}), + ): + instance = MediaPlayerProxy.get_instance() + assert isinstance(instance, MPVMediaPlayer) + + def test_get_instance_returns_mpv_for_pi4_64(reset_media_proxy: None) -> None: MediaPlayerProxy.INSTANCE = None with ( @@ -624,71 +771,100 @@ def _rotated_mpv_settings(rotation: int) -> Any: return mock +@pytest.mark.parametrize('device_type', ['x86', 'arm64', 'pi5']) +@pytest.mark.parametrize('rotation', [0, 90, 180, 270]) +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='', +) @patch( 'anthias_viewer.media_player._detect_hdmi_audio_device', return_value='sysdefault:CARD=vc4hdmi0', ) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_mpv_passes_video_rotate_on_pi( - mock_popen: Any, _mock_detect: Any +def test_mpv_never_passes_video_rotate_under_cage( + mock_popen: Any, + _mock_detect: Any, + _mock_probe: Any, + rotation: int, + device_type: str, ) -> None: - """On --vo=drm the framebuffer is written direct; rotation has to - happen inside mpv.""" + """x86 / arm64 / pi5 run under cage and inherit the compositor + transform via wlr-randr (issue #2856 — driven from + src/anthias_viewer/__init__.py). Passing --video-rotate to mpv + would double-rotate, so MPVMediaPlayer must never add it on + those boards.""" player = MPVMediaPlayer() + # Patch get_device_type alongside DEVICE_TYPE so + # get_alsa_audio_device() takes a deterministic branch on the + # host — patching only DEVICE_TYPE while leaving get_device_type + # at the fixture default ('pi4') would route x86/arm64 through + # _detect_hdmi_audio_device() and stat /sys/class/drm. + audio_device_type = device_type if device_type == 'pi5' else 'x86' with ( patch( 'anthias_viewer.media_player.settings', - _rotated_mpv_settings(180), + _rotated_mpv_settings(rotation), ), patch( 'anthias_viewer.media_player.get_device_type', - return_value='pi5', + return_value=audio_device_type, ), - patch.dict('os.environ', {'DEVICE_TYPE': 'pi5'}), + patch.dict('os.environ', {'DEVICE_TYPE': device_type}), ): player.set_asset('file:///test/video.mp4', 30) player.play() args, _ = mock_popen.call_args - assert '--video-rotate=180' in args[0] + assert not any(arg.startswith('--video-rotate') for arg in args[0]) +@pytest.mark.parametrize('rotation', [90, 180, 270]) +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='', +) +@patch( + 'anthias_viewer.media_player._detect_hdmi_audio_device', + return_value='sysdefault:CARD=vc4hdmi0', +) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_mpv_skips_video_rotate_on_x86(mock_popen: Any) -> None: - """x86 inherits the compositor transform via wayland; double- - rotating in mpv would undo wlr-randr's work.""" +def test_mpv_passes_video_rotate_on_pi4_64( + mock_popen: Any, _mock_detect: Any, _mock_probe: Any, rotation: int +) -> None: + """Pi 4 stays on Qt linuxfb (no cage), so there's no compositor + transform to inherit — mpv has to apply rotation itself.""" player = MPVMediaPlayer() - # Patch get_device_type to 'x86' so get_alsa_audio_device() takes - # the HID-card fallback branch — patching it to 'pi5' while - # DEVICE_TYPE=x86 would route through _detect_hdmi_audio_device() - # and stat /sys/class/drm, making the test depend on the host. with ( patch( 'anthias_viewer.media_player.settings', - _rotated_mpv_settings(90), + _rotated_mpv_settings(rotation), ), patch( 'anthias_viewer.media_player.get_device_type', - return_value='x86', + return_value='pi5', ), - patch.dict('os.environ', {'DEVICE_TYPE': 'x86'}), + patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}), ): player.set_asset('file:///test/video.mp4', 30) player.play() args, _ = mock_popen.call_args - assert not any(arg.startswith('--video-rotate') for arg in args[0]) + assert f'--video-rotate={rotation}' in args[0] @patch( 'anthias_viewer.media_player._detect_hdmi_audio_device', return_value='sysdefault:CARD=vc4hdmi0', ) +@patch( + 'anthias_viewer.media_player._probe_video_codec', + return_value='', +) @patch('anthias_viewer.media_player.subprocess.Popen') -def test_mpv_no_video_rotate_at_zero( - mock_popen: Any, _mock_detect: Any +def test_mpv_skips_video_rotate_at_zero_on_pi4_64( + mock_popen: Any, _mock_probe: Any, _mock_detect: Any ) -> None: - """The default-orientation case must NOT add --video-rotate=0 — - keeps the CLI surface unchanged for the 99% of operators who never - touch the dropdown, matching the existing arg-list test.""" + """0° must NOT emit --video-rotate=0 — keeps the CLI surface + unchanged for the 99% of operators who never touch the dropdown.""" player = MPVMediaPlayer() with ( patch( @@ -699,7 +875,7 @@ def test_mpv_no_video_rotate_at_zero( 'anthias_viewer.media_player.get_device_type', return_value='pi5', ), - patch.dict('os.environ', {'DEVICE_TYPE': 'pi5'}), + patch.dict('os.environ', {'DEVICE_TYPE': 'pi4-64'}), ): player.set_asset('file:///test/video.mp4', 30) player.play() @@ -716,3 +892,34 @@ def test_proxy_reset_clears_cached_instance(reset_media_proxy: None) -> None: MediaPlayerProxy.reset() assert MediaPlayerProxy.INSTANCE is None fake.stop.assert_called_once() + + +def test_pi_hwdec_dispatch_matches_upload_gate() -> None: + """The upload gate (``processing._HW_DECODE_VIDEO_CODECS``) and + the viewer's per-codec mpv dispatch (``_PI_HWDEC_BY_CODEC``) must + not drift. If the gate accepts a codec on a board, the viewer + must have an explicit hwdec entry for it — otherwise the operator + uploads a clip that passes validation, then plays back through + ``auto-copy`` and quietly software-decodes, defeating the gate. + + The reverse direction (viewer claims to hardware-decode a codec + the gate rejects) is just as bad — the viewer path becomes dead + code while the gate refuses every upload that would exercise it. + + Boards present in the gate but absent from ``_PI_HWDEC_BY_CODEC`` + (``pi2``, ``pi3``, ``x86``) are intentional: Pi 2/3 use VLC, x86 + falls through to mpv ``--hwdec=auto-copy`` which picks vaapi-copy + on Intel iGPUs. Those paths don't need an explicit table entry.""" + from anthias_server.processing import _HW_DECODE_VIDEO_CODECS + from anthias_viewer.media_player import _PI_HWDEC_BY_CODEC + + for board, viewer_codecs in _PI_HWDEC_BY_CODEC.items(): + assert board in _HW_DECODE_VIDEO_CODECS, ( + f'viewer dispatches {board!r} but the upload gate has no ' + f'entry — every upload to that board would be rejected.' + ) + gate_codecs = _HW_DECODE_VIDEO_CODECS[board] + assert frozenset(viewer_codecs) == gate_codecs, ( + f'codec mismatch for {board!r}: ' + f'viewer={sorted(viewer_codecs)} gate={sorted(gate_codecs)}' + ) diff --git a/tests/test_processing.py b/tests/test_processing.py index ba65ccc76..ac51a747d 100644 --- a/tests/test_processing.py +++ b/tests/test_processing.py @@ -6,12 +6,10 @@ ``NORMALIZE_IMAGE_EXTS`` (HEIC / HEIF / TIFF / BMP / ICO / TGA / JPEG 2000 family / AVIF) → lossless WebP. JPEG / PNG / WebP / GIF / SVG short-circuit through the no-op branch. -* ``normalize_video_asset`` — passthrough or transcode driven by an - ffprobe call against the source. Transcode target depends on the - board profile resolved from ``DEVICE_TYPE``: libx264 on legacy Pi - 2 / Pi 3 (no HEVC hardware), libx265 + ``-tag:v hvc1`` on Pi 4-64 - / Pi 5 / x86. The grid lives in ``processing._BOARD_PROFILES`` - and is exercised end-to-end by the per-board parametrised tests. +* ``normalize_video_asset`` — runs ffprobe and writes codec / dims / + fps / audio codec / container / duration into ``metadata``. The + asset file is never rewritten; the operator's UI uses the metadata + fields to identify clips the board can't decode in hardware. Fixtures are generated programmatically (Pillow + ffmpeg) so the test suite is self-contained — no checked-in binary blobs to drift, and @@ -600,317 +598,291 @@ def test_set_processing_error_writes_metadata(asset_dir: str) -> None: # --------------------------------------------------------------------------- +@pytest.mark.django_db +def test_video_missing_file_raises_filenotfound(asset_dir: str) -> None: + src = path.join(asset_dir, 'gone.mp4') + asset = _make_processing_asset('vid-gone', src, mimetype='video') + with mock.patch.object(processing, '_notify'): + with pytest.raises(FileNotFoundError): + processing._run_video_normalisation(asset) + + @pytest_ffmpeg @pytest.mark.django_db -def test_video_h264_mp4_passes_through(asset_dir: str) -> None: - """The bread-and-butter case: an H.264 MP4 with AAC audio. ffmpeg - is *not* called; the row gets duration + metadata + is_processing - cleared, file untouched on disk.""" +def test_video_supported_codec_writes_metadata_and_clears_processing( + asset_dir: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """H.264 upload on a Pi 4 (which HW-decodes H.264) is accepted: + the row gets codec / dims / fps written into ``metadata`` and + ``is_processing`` is cleared.""" + monkeypatch.setenv('DEVICE_TYPE', 'pi4-64') src = path.join(asset_dir, 'sample.mp4') _make_video(src, codec='libx264', container='mp4', audio='aac') - asset = _make_processing_asset('vid-h264', src, mimetype='video') + asset = _make_processing_asset('vid-h264-pi4', src, mimetype='video') - pre_size = os.stat(src).st_size - with mock.patch.object(processing, '_notify') as notify: + with mock.patch.object(processing, '_notify'): processing._run_video_normalisation(asset) asset.refresh_from_db() - assert asset.uri == src # passthrough — no rename assert asset.is_processing is False - assert asset.metadata['original_ext'] == '.mp4' - assert asset.metadata['transcoded'] is False - # Duration has been probed in (>= 1 second floor). - assert asset.duration is not None and asset.duration >= 1 - assert os.stat(src).st_size == pre_size # bytes untouched - notify.assert_called_once_with('vid-h264') + assert asset.metadata.get('video_codec') == 'h264' + assert asset.metadata.get('video_width') == 32 + assert asset.metadata.get('video_height') == 32 + # The asset file itself is untouched — no transcode. + assert path.exists(src) @pytest_ffmpeg @pytest.mark.django_db -@pytest.mark.parametrize( - ('codec', 'ext', 'container'), - [ - ('libx264', '.mkv', 'matroska'), - ('libx264', '.mov', 'mov'), - ('libx265', '.mp4', 'mp4'), - ('libx265', '.mkv', 'matroska'), - ], -) -def test_video_passthrough_for_h264_or_hevc_in_known_containers( - asset_dir: str, - codec: str, - ext: str, - container: str, - monkeypatch: pytest.MonkeyPatch, +def test_video_unsupported_codec_raises_with_ffmpeg_recipe( + asset_dir: str, monkeypatch: pytest.MonkeyPatch ) -> None: - """H.264 and HEVC in any of the accepted containers passes - through *on a board profile that supports HEVC*. Pin - ``DEVICE_TYPE=pi5`` so the libx265-source rows hit passthrough - rather than getting transcoded back down to H.264 by the - default profile.""" + """An H.264 upload on a Pi 5 (HEVC only — Pi 5's mpv has no v4l2- + request H.264 hwdec) is rejected. The exception's message names + the rejected codec and supported set; its ``recipe`` attribute + carries an ffmpeg command pre-filled with the upload's filename + (taken from ``metadata.upload_name``, which the upload view + stashes at create time) so the operator can copy-paste it + verbatim.""" monkeypatch.setenv('DEVICE_TYPE', 'pi5') - src = path.join(asset_dir, f'sample{ext}') - _make_video(src, codec=codec, container=container, audio='aac') - asset = _make_processing_asset('vid-pass', src, mimetype='video') + src = path.join(asset_dir, 'sample.mp4') + _make_video(src, codec='libx264', container='mp4', audio='aac') + asset = _make_processing_asset( + 'vid-h264-pi5', + src, + mimetype='video', + metadata={'upload_name': 'beach-clip.mp4'}, + ) with mock.patch.object(processing, '_notify'): - processing._run_video_normalisation(asset) - - asset.refresh_from_db() - assert asset.metadata['transcoded'] is False - assert asset.uri == src - + with pytest.raises(processing.UnsupportedVideoCodecError) as excinfo: + processing._run_video_normalisation(asset) -@pytest_ffmpeg -@pytest.mark.django_db -def test_video_silent_passes_through(asset_dir: str) -> None: - """A muted clip (no audio stream at all) must passthrough — the - audio_codec=='none' branch is the third leg of - _video_can_passthrough and the easiest to regress.""" - src = path.join(asset_dir, 'silent.mp4') - _make_video(src, codec='libx264', container='mp4', audio=None) - asset = _make_processing_asset('vid-silent', src, mimetype='video') - - with mock.patch.object(processing, '_notify'): - processing._run_video_normalisation(asset) + import shlex as _shlex + msg = str(excinfo.value) + assert "'h264'" in msg + assert 'hevc' in msg + # Recipe is on the exception, not in the message body. + recipe = excinfo.value.recipe + assert 'libx265' in recipe # Pi 5 supports HEVC only. + assert '-tag:v hvc1' in recipe + # The upload's filename appears in the recipe's input slot — + # operator can copy and paste it without hand-editing INPUT. + tokens = _shlex.split(recipe) + assert tokens[1] == '-i' + assert tokens[2] == 'beach-clip.mp4' + # Output filename carries a ``.hevc.`` suffix so the recipe + # doesn't ask the operator to overwrite their source file. + assert tokens[-1] == 'beach-clip.hevc.mp4' + + # Metadata was still written so the operator can see *what* they + # uploaded next to the error message in the asset list. asset.refresh_from_db() - assert asset.metadata['transcoded'] is False + assert asset.metadata.get('video_codec') == 'h264' + assert asset.metadata.get('video_width') == 32 @pytest_ffmpeg @pytest.mark.django_db -@pytest.mark.parametrize( - ('codec', 'ext', 'container', 'extra'), - [ - # MPEG-2 in an MPEG-PS container — common camcorder dump. - ('mpeg2video', '.mpg', 'mpeg', ()), - # Motion JPEG — exotic but ffmpeg-supported. - ('mjpeg', '.avi', 'avi', ('-q:v', '5')), - ], -) -def test_video_exotic_codec_transcodes_to_h264_mp4( - asset_dir: str, - codec: str, - ext: str, - container: str, - extra: tuple[str, ...], +def test_video_unsupported_codec_recipe_falls_back_to_upload_placeholder( + asset_dir: str, monkeypatch: pytest.MonkeyPatch ) -> None: - """Codecs outside the passthrough set become H.264 + AAC MP4. The - output filename ends in .mp4 regardless of the source extension; - the source file is removed once the .mp4 is in place.""" - src = path.join(asset_dir, f'fixture{ext}') - _make_video( - src, codec=codec, container=container, audio='mp2', extra_args=extra - ) - asset = _make_processing_asset('vid-tc', src, mimetype='video') + """When the row has no ``metadata.upload_name`` (YouTube + downloads, pre-rebrand rows), the recipe uses a stable + ``upload`` placeholder so the operator still sees the + correct input extension to substitute.""" + monkeypatch.setenv('DEVICE_TYPE', 'pi5') + src = path.join(asset_dir, 'noname.mp4') + _make_video(src, codec='libx264', container='mp4', audio='aac') + asset = _make_processing_asset('vid-noname', src, mimetype='video') - with mock.patch.object(processing, '_notify') as notify: - processing._run_video_normalisation(asset) + with mock.patch.object(processing, '_notify'): + with pytest.raises(processing.UnsupportedVideoCodecError) as excinfo: + processing._run_video_normalisation(asset) - asset.refresh_from_db() - final_uri = path.join(asset_dir, 'fixture.mp4') - assert asset.uri == final_uri - assert path.isfile(final_uri) - assert not path.exists(src), 'original must be removed after transcode' - assert asset.metadata['transcoded'] is True - assert asset.metadata['original_ext'] == ext - assert asset.is_processing is False - notify.assert_called_once_with('vid-tc') - - # Verify the output is actually H.264 in a passthrough-eligible - # container — not just an extension-rename of the source. The - # container check uses the passthrough set rather than asserting - # ``container == 'mp4'`` directly because ffprobe's - # format.format_name reports a comma-joined synonym list for MP4 - # files (e.g. ``mov,mp4,m4a,3gp,3g2,mj2``); _ffprobe_summary - # picks whichever token first matches the passthrough set, and - # the exact pick is implementation detail. - summary = processing._ffprobe_summary(final_uri) - assert summary['video_codec'] == 'h264' - assert summary['container'] in processing._PASSTHROUGH_CONTAINERS + import shlex as _shlex + + recipe = excinfo.value.recipe + tokens = _shlex.split(recipe) + assert tokens[1] == '-i' + assert tokens[2] == 'upload.mp4' + assert tokens[-1] == 'upload.hevc.mp4' @pytest_ffmpeg @pytest.mark.django_db -def test_video_non_h264_mp4_is_transcoded_in_place(asset_dir: str) -> None: - """An MP4-container with a non-passthrough codec needs a transcode - even though the extension is already .mp4. Test the staging - rename: source must NOT be truncated mid-read by the output going - to the same path. Output ends up at the same `.mp4` URI.""" - src = path.join(asset_dir, 'fixture.mp4') - # MPEG-4 Part 2 (xvid-style). Neither h264 nor hevc → must - # transcode despite landing in mp4. - _make_video(src, codec='mpeg4', container='mp4', audio='aac') - pre_inode = os.stat(src).st_ino - asset = _make_processing_asset('vid-mpeg4', src, mimetype='video') +def test_video_unsupported_codec_h264_board_recipe( + asset_dir: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """A board that supports H.264 (Pi 4) gets a libx264 recipe — + libx264 is significantly faster than libx265 for the operator.""" + monkeypatch.setenv('DEVICE_TYPE', 'pi4-64') + src = path.join(asset_dir, 'sample.mpg') + _make_video(src, codec='mpeg2video', container='mpeg', audio=None) + asset = _make_processing_asset('vid-mpeg2', src, mimetype='video') with mock.patch.object(processing, '_notify'): - processing._run_video_normalisation(asset) + with pytest.raises(processing.UnsupportedVideoCodecError) as excinfo: + processing._run_video_normalisation(asset) - asset.refresh_from_db() - assert asset.uri == src - summary = processing._ffprobe_summary(src) - assert summary['video_codec'] == 'h264' - # Post-transcode the on-disk inode must differ — the staging - # rename replaced the original; we did not in-place truncate. - assert os.stat(src).st_ino != pre_inode + recipe = excinfo.value.recipe + assert 'libx264' in recipe + assert 'libx265' not in recipe -@pytest.mark.django_db -def test_video_missing_file_raises_filenotfound(asset_dir: str) -> None: - src = path.join(asset_dir, 'gone.mp4') - asset = _make_processing_asset('vid-gone', src, mimetype='video') - with mock.patch.object(processing, '_notify'): - with pytest.raises(FileNotFoundError): - processing._run_video_normalisation(asset) +@pytest.mark.parametrize( + 'filename', + [ + "O'Brien.mp4", + 'two words.mov', + 'evil; rm -rf $HOME.mp4', + 'tick`uname`.mp4', + 'sub$(whoami).mp4', + ], +) +def test_ffmpeg_recipe_quotes_hostile_filenames(filename: str) -> None: + """``_ffmpeg_reencode_recipe`` must round-trip any filename through + ``shlex`` so a user-supplied ``upload_name`` can't break out of the + recipe's quoting and inject commands the operator copy-pastes. + + Round-trip means: ``shlex.split(recipe)`` recovers the *original* + filename byte-for-byte in the input slot. If the recipe still + interpolated raw (the pre-fix ``f"'{filename}'"`` path), the + embedded quote / metachar would either truncate the token or shell- + interpret on paste.""" + import shlex as _shlex + + recipe = processing._ffmpeg_reencode_recipe(frozenset({'h264'}), filename) + tokens = _shlex.split(recipe) + # ffmpeg -i -c:v libx264 ... + assert tokens[0] == 'ffmpeg' + assert tokens[1] == '-i' + assert tokens[2] == filename + # Output filename ends with .h264.mp4 and is the recipe's last token. + assert tokens[-1].endswith('.h264.mp4') +@pytest_ffmpeg @pytest.mark.django_db -def test_video_ffprobe_failure_falls_through_to_transcode( - asset_dir: str, +def test_video_unknown_codec_is_rejected( + asset_dir: str, monkeypatch: pytest.MonkeyPatch ) -> None: - """A probe that crashes (corrupt header) returns 'unknown' for - every dimension; _video_can_passthrough rejects unknowns so the - code falls through to transcode. We mock ffprobe to verify the - branch wires up — running the real probe on a synthetic corrupt - file is non-deterministic across ffprobe versions.""" - src = path.join(asset_dir, 'broken.mp4') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 32) - asset = _make_processing_asset('vid-broken', src, mimetype='video') + """ffprobe failure (codec reported as 'unknown') must reject the + upload — we won't pass through a clip we can't certify against + the board's HW decode set.""" + monkeypatch.setenv('DEVICE_TYPE', 'pi4-64') + src = path.join(asset_dir, 'sample.mp4') + _make_video(src, codec='libx264', container='mp4', audio='aac') + asset = _make_processing_asset('vid-unknown', src, mimetype='video') fake_summary = { 'container': 'unknown', 'video_codec': 'unknown', + 'video_pixels': None, + 'video_width': None, + 'video_height': None, + 'video_fps': None, 'audio_codec': 'unknown', 'duration_seconds': None, } - - def fake_transcode(_in: str, out: str, _profile: Any = None) -> None: - with open(out, 'wb') as fh: - fh.write(b'\x00\x00\x00\x18ftypmp42') # 24-byte stub - - def fake_probe_post(uri: str) -> int | None: - return 5 # mocked duration - with ( mock.patch.object(processing, '_notify'), mock.patch.object( processing, '_ffprobe_summary', return_value=fake_summary ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=fake_transcode - ), - mock.patch.object( - processing, - '_resolve_duration_seconds', - side_effect=fake_probe_post, - ), + pytest.raises(processing.UnsupportedVideoCodecError) as excinfo, ): processing._run_video_normalisation(asset) - asset.refresh_from_db() - assert asset.metadata['transcoded'] is True - assert asset.duration == 5 + assert 'unknown' in str(excinfo.value) @pytest.mark.django_db -def test_video_ffmpeg_timeout_cleans_staging(asset_dir: str) -> None: - """ffmpeg time-limit overrun: staging file removed, RuntimeError - raised so on_failure clears is_processing. Mocking the transcode - helper directly because reproducing a real time-limit kill in a - unit test is brittle (depends on subprocess scheduling).""" - src = path.join(asset_dir, 'bigfile.mov') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 256) - asset = _make_processing_asset('vid-timeout', src, mimetype='video') - - summary = { - 'container': 'mov', - 'video_codec': 'prores', # not passthrough +def test_video_arm64_catch_all_rejects_everything( + asset_dir: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """The catch-all ``arm64`` DEVICE_TYPE has no entry in the HW + decode map (an unknown aarch64 SBC isn't guaranteed to expose a + v4l2-request decoder mpv can address). Without a host_agent + subtype publish, every video upload is rejected — operator has + to install a board-specific image to get HW decode.""" + monkeypatch.setenv('DEVICE_TYPE', 'arm64') + src = path.join(asset_dir, 'sample.mp4') + # Create an empty placeholder file so the FileNotFoundError check + # passes; we mock ffprobe below. + with open(src, 'wb') as f: + f.write(b'\x00') + asset = _make_processing_asset('vid-arm64', src, mimetype='video') + + fake_summary = { + 'container': 'mp4', + 'video_codec': 'h264', + 'video_pixels': 32 * 32, + 'video_width': 32, + 'video_height': 32, + 'video_fps': 10.0, 'audio_codec': 'aac', + 'duration_seconds': 1, } - - def explode(_in: str, staging: str, _profile: Any = None) -> None: - # Half-write the staging file so the cleanup branch has - # something to remove — proves we don't leak orphans. - with open(staging, 'wb') as fh: - fh.write(b'partial') - raise sh.TimeoutException( - exit_code=124, - full_cmd='ffmpeg ...', - ) - with ( mock.patch.object(processing, '_notify'), mock.patch.object( - processing, '_ffprobe_summary', return_value=summary + processing, '_ffprobe_summary', return_value=fake_summary ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=explode + # Subtype absent — Redis returns None. + mock.patch( + 'anthias_common.board.get_board_subtype', return_value=None ), + pytest.raises(processing.UnsupportedVideoCodecError) as excinfo, ): - with pytest.raises(RuntimeError): - processing._run_video_normalisation(asset) + processing._run_video_normalisation(asset) - # Staging file was cleaned up. - leftover = [ - n for n in os.listdir(asset_dir) if n.startswith('bigfile.mp4') - ] - assert not leftover, f'staging leftover: {leftover}' + msg = str(excinfo.value) + # Catch-all branch must explain the board-subtype gap rather + # than the misleading "Supported: none." that earlier revisions + # surfaced. + assert 'subtype' in msg.lower() + assert 'board-specific image' in msg.lower() @pytest.mark.django_db -def test_video_ffmpeg_error_cleans_staging(asset_dir: str) -> None: - """Same shape as the timeout test but for a non-zero ffmpeg - exit. RuntimeError must include stderr so the operator gets a - diagnostic in metadata.error_message.""" - src = path.join(asset_dir, 'bad.avi') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 16) - asset = _make_processing_asset('vid-fail', src, mimetype='video') - - summary = { - 'container': 'avi', - 'video_codec': 'cinepak', - 'audio_codec': 'pcm_s16le', - } - - def explode(_in: str, staging: str, _profile: Any = None) -> None: - with open(staging, 'wb') as fh: - fh.write(b'') - # ``ErrorReturnCode`` is the abstract parent — sh exports - # numeric subclasses (ErrorReturnCode_1, ..._127) for each - # exit code. The processing code catches the parent class so - # the test can raise any subclass. - raise sh.ErrorReturnCode_1( - full_cmd='ffmpeg ...', - stdout=b'', - stderr=b'Invalid data found', - truncate=False, - ) +def test_video_arm64_with_rockpi4_subtype_accepts_h264( + asset_dir: str, monkeypatch: pytest.MonkeyPatch +) -> None: + """A Rock Pi 4 running the catch-all arm64 image gets its + ``{h264, hevc}`` set once ``host:board_subtype=rockpi4`` is + published by anthias_host_agent.""" + monkeypatch.setenv('DEVICE_TYPE', 'arm64') + src = path.join(asset_dir, 'sample.mp4') + with open(src, 'wb') as f: + f.write(b'\x00') + asset = _make_processing_asset('vid-rockpi', src, mimetype='video') + fake_summary = { + 'container': 'mp4', + 'video_codec': 'h264', + 'video_pixels': 32 * 32, + 'video_width': 32, + 'video_height': 32, + 'video_fps': 10.0, + 'audio_codec': 'aac', + 'duration_seconds': 1, + } with ( mock.patch.object(processing, '_notify'), mock.patch.object( - processing, '_ffprobe_summary', return_value=summary + processing, '_ffprobe_summary', return_value=fake_summary ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=explode + mock.patch( + 'anthias_common.board.get_board_subtype', return_value='rockpi4' ), ): - with pytest.raises(RuntimeError) as excinfo: - processing._run_video_normalisation(asset) + processing._run_video_normalisation(asset) - msg = str(excinfo.value) - assert 'Invalid data found' in msg - # The error message goes straight into metadata.error_message - # which renders on the operator-facing "Failed" pill — must NOT - # contain a Python bytes repr (``b'...'``) wrapper. - assert "b'Invalid" not in msg, ( - 'stderr should be decoded for operator display' - ) + asset.refresh_from_db() + assert asset.is_processing is False + assert asset.metadata.get('video_codec') == 'h264' def test_format_subprocess_stderr_decodes_and_trims() -> None: @@ -964,89 +936,6 @@ def test_format_subprocess_stderr_decodes_and_trims() -> None: assert processing._format_subprocess_stderr(exc) == '' -@pytest.mark.django_db -def test_video_zero_byte_output_fails_clean(asset_dir: str) -> None: - """ffmpeg sometimes returns exit 0 but produces an empty file - (broken stream, codec mismatch the syntax would have rejected - in newer builds). The task must reject the empty output and - raise — never advertise a 0-byte .mp4 as ready.""" - src = path.join(asset_dir, 'odd.mov') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 16) - asset = _make_processing_asset('vid-empty', src, mimetype='video') - - summary = { - 'container': 'mov', - 'video_codec': 'prores', - 'audio_codec': 'aac', - } - - def empty_transcode(_in: str, staging: str, _profile: Any = None) -> None: - with open(staging, 'wb') as fh: - fh.write(b'') - - with ( - mock.patch.object(processing, '_notify'), - mock.patch.object( - processing, '_ffprobe_summary', return_value=summary - ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=empty_transcode - ), - ): - with pytest.raises(RuntimeError, match='no output'): - processing._run_video_normalisation(asset) - - # The empty staging file must be removed too, not just the - # error raised — otherwise cleanup() would have to GC it - # later via the orphan-file sweep. Same contract as the - # timeout/error branches above. - leftover = [n for n in os.listdir(asset_dir) if 'staging' in n] - assert not leftover, f'staging leftover after empty output: {leftover}' - - -@pytest.mark.django_db -def test_video_rename_failure_cleans_staging(asset_dir: str) -> None: - """Video pipeline mirrors the image-pipeline contract: an OSError - on the post-transcode ``os.replace(staging, final_uri)`` (disk - full, permissions, cross-device) drops the .staging.mp4 file - before propagating.""" - src = path.join(asset_dir, 'odd.mov') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 16) - asset = _make_processing_asset('vid-rename-fail', src, mimetype='video') - - summary = { - 'container': 'mov', - 'video_codec': 'prores', - 'audio_codec': 'aac', - } - - def good_transcode(_in: str, staging: str, _profile: Any = None) -> None: - with open(staging, 'wb') as fh: - fh.write(b'\x00\x00\x00\x18ftypmp42') - - def boom(staging: str, final_uri: str) -> None: - assert path.isfile(staging), 'precondition: staging must exist' - raise OSError('simulated rename failure') - - with ( - mock.patch.object(processing, '_notify'), - mock.patch.object( - processing, '_ffprobe_summary', return_value=summary - ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=good_transcode - ), - mock.patch('anthias_server.processing.os.replace', side_effect=boom), - ): - with pytest.raises(OSError, match='rename failure'): - processing._run_video_normalisation(asset) - - leftover = [n for n in os.listdir(asset_dir) if n.endswith('.staging.mp4')] - assert not leftover, f'video staging leftover after rename: {leftover}' - - # --------------------------------------------------------------------------- # ffprobe summary parsing — tested independently of the runner # --------------------------------------------------------------------------- @@ -1087,13 +976,58 @@ def test_ffprobe_summary_handles_no_audio_track() -> None: assert summary['audio_codec'] == 'none' +@pytest.mark.parametrize( + ('r_frame_rate', 'expected_fps'), + [ + # Integer rates land cleanly. + ('30/1', 30.0), + ('60/1', 60.0), + ('25/1', 25.0), + # NTSC drop-frame: 30000/1001 ≈ 29.97. + ('30000/1001', 29.97002997002997), + # 60000/1001 ≈ 59.94 (NTSC 60). + ('60000/1001', 59.94005994005994), + # Garbage values collapse to None so the envelope cap + # treats the source as "we can't tell" and skips the fps + # gate — codec / resolution gates still fire. + ('bogus', None), + ('60', None), # no slash → no rational, drop to None + ('0/0', None), # denominator 0 → no fps + ], +) +def test_ffprobe_summary_parses_video_fps( + r_frame_rate: str, expected_fps: float | None +) -> None: + """``video_fps`` is the average frame rate parsed from + ffprobe's ``r_frame_rate`` rational. The envelope transcode + uses it to decide when to emit ``-r envelope.max_fps`` — only + when source fps > cap. Garbage / zero-denominator → ``None``.""" + fake = { + 'format': {}, + 'streams': [ + { + 'codec_type': 'video', + 'codec_name': 'h264', + 'r_frame_rate': r_frame_rate, + }, + ], + } + with mock.patch.object(processing, '_ffprobe_streams', return_value=fake): + summary = processing._ffprobe_summary('fixture.mp4') + if expected_fps is None: + assert summary['video_fps'] is None + else: + assert summary['video_fps'] == pytest.approx(expected_fps) + + def test_ffprobe_summary_prefers_format_name_over_filename_extension() -> None: """Defensive: ffprobe-reported ``format.format_name`` beats the - filename. A ``.bin`` file that's actually an MP4 must classify - as passthrough-eligible — and a ``.mp4`` file whose bytes are - actually a non-passthrough format (e.g. ``avi``) must classify - out of the passthrough set despite the misleading extension.""" - # MP4 bytes hidden behind an arbitrary extension. + filename. A ``.bin`` file that's actually an MP4 reports the + canonical container token from ffprobe; a ``.mp4`` file whose + bytes are a non-mp4 format reports the format token verbatim, + not the misleading extension.""" + # MP4 bytes hidden behind an arbitrary extension — the first + # token in ffprobe's synonym list is the canonical name. mp4_format_name = 'mov,mp4,m4a,3gp,3g2,mj2' fake = { 'format': {'format_name': mp4_format_name}, @@ -1101,14 +1035,9 @@ def test_ffprobe_summary_prefers_format_name_over_filename_extension() -> None: } with mock.patch.object(processing, '_ffprobe_streams', return_value=fake): summary = processing._ffprobe_summary('fixture.bin') - # The picked token matches the passthrough set. - assert summary['container'] in processing._PASSTHROUGH_CONTAINERS - - # AVI bytes hidden behind a `.mp4` filename — must NOT pass - # through. avi is intentionally in the passthrough list (h264 - # in avi is fine), but if format.format_name returns just - # 'foo' (made up, not in our set) we report that token verbatim - # so the caller falls through to transcode. + assert summary['container'] == 'mov' + + # Made-up format name — reported verbatim, no extension fallback. fake = { 'format': {'format_name': 'unsupported_format'}, 'streams': [{'codec_type': 'video', 'codec_name': 'h264'}], @@ -1116,57 +1045,6 @@ def test_ffprobe_summary_prefers_format_name_over_filename_extension() -> None: with mock.patch.object(processing, '_ffprobe_streams', return_value=fake): summary = processing._ffprobe_summary('fixture.mp4') assert summary['container'] == 'unsupported_format' - assert summary['container'] not in processing._PASSTHROUGH_CONTAINERS - - -@pytest.mark.parametrize( - ('format_name', 'description'), - [ - # Real ffprobe format_name strings observed for each container - # in the passthrough set. The decision must classify them all - # as eligible — without ``mpegts`` / ``matroska`` in the set, - # an MPEG-TS or MKV upload would force an unnecessary transcode - # despite both being playable on every Anthias-supported board. - ('mov,mp4,m4a,3gp,3g2,mj2', '.mp4 / .m4v / .mov'), - ('matroska,webm', '.mkv / .webm'), - ('mpegts', '.ts'), - ('mpeg', '.mpg / .mpeg'), - ('flv', '.flv'), - ('avi', '.avi'), - ], -) -def test_passthrough_containers_match_real_ffprobe_format_names( - format_name: str, description: str -) -> None: - """Every container that's listed in ``_PASSTHROUGH_CONTAINERS`` as - a "we accept this" must actually match what ffprobe writes for - real files of that container — not just the file's extension. - - Regression: ffprobe reports ``mpegts`` for .ts (not ``ts``) and - ``matroska`` for .mkv (not ``mkv``). The passthrough set used to - carry only the short extension labels, so MPEG-TS uploads were - being unnecessarily re-encoded. Adding the canonical ffprobe - names to the set keeps the decision consistent between the - extension-fallback and format_name-driven detection paths. - """ - fake = { - 'format': {'format_name': format_name}, - 'streams': [ - {'codec_type': 'video', 'codec_name': 'h264'}, - {'codec_type': 'audio', 'codec_name': 'aac'}, - ], - } - with mock.patch.object(processing, '_ffprobe_streams', return_value=fake): - summary = processing._ffprobe_summary('fixture.unused') - assert summary['container'] in processing._PASSTHROUGH_CONTAINERS, ( - f'{description}: ffprobe format_name={format_name!r} resolved to ' - f'{summary["container"]!r} which is not in _PASSTHROUGH_CONTAINERS' - ) - pi5 = processing._BOARD_PROFILES['pi5'] - assert processing._video_can_passthrough(summary, pi5), ( - f'{description}: passthrough check rejected a real-ffprobe-name ' - f'container; would force an unnecessary transcode' - ) def test_ffprobe_summary_falls_back_to_extension_when_format_missing() -> None: @@ -1198,6 +1076,10 @@ def test_ffprobe_summary_handles_probe_failure() -> None: assert summary == { 'container': 'unknown', 'video_codec': 'unknown', + 'video_pixels': None, + 'video_width': None, + 'video_height': None, + 'video_fps': None, 'audio_codec': 'unknown', 'duration_seconds': None, } @@ -1247,57 +1129,6 @@ def test_ffprobe_summary_extracts_duration_from_probe_payload() -> None: assert summary['duration_seconds'] is None -@pytest.mark.django_db -def test_video_passthrough_uses_summary_duration_no_second_probe( - asset_dir: str, monkeypatch: pytest.MonkeyPatch -) -> None: - """The passthrough branch must reuse the duration the summary - already extracted; calling ``get_video_duration`` (which would - re-shell ffprobe) is a regression. Asserts via mock-not-called.""" - monkeypatch.setenv('DEVICE_TYPE', 'pi5') - src = path.join(asset_dir, 'clip.mp4') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 64) - asset = _make_processing_asset('vid-no-2nd-probe', src, mimetype='video') - - summary = { - 'container': 'mp4', - 'video_codec': 'h264', - 'audio_codec': 'aac', - 'duration_seconds': 42, - } - with ( - mock.patch.object( - processing, '_ffprobe_summary', return_value=summary - ), - mock.patch.object(processing, 'get_video_duration') as get_dur, - mock.patch.object(processing, '_notify'), - ): - processing._run_video_normalisation(asset) - - asset.refresh_from_db() - assert asset.duration == 42 - # Crucially: the second ffprobe shell never happened. - get_dur.assert_not_called() - - -def test_resolve_duration_seconds_swallows_probe_exceptions() -> None: - """``get_video_duration`` raises on ffprobe errors. After a - successful transcode the row is otherwise ready to play; failing - the entire task because the post-transcode duration probe stumbled - would lose all the work. Helper must catch and return None so the - runner just skips the duration update and lets the operator edit - manually.""" - with mock.patch.object( - processing, - 'get_video_duration', - side_effect=Exception('Bad video format'), - ): - # Should NOT raise. - result = processing._resolve_duration_seconds('clip.mp4') - assert result is None - - def test_format_subprocess_stderr_byte_trim_handles_multibyte_utf8() -> None: """The trim is documented as a byte limit; multibyte characters in the keep window must not push the decoded string over the @@ -1335,263 +1166,15 @@ def test_ffprobe_summary_handles_missing_ffprobe_binary() -> None: assert summary == { 'container': 'unknown', 'video_codec': 'unknown', + 'video_pixels': None, + 'video_width': None, + 'video_height': None, + 'video_fps': None, 'audio_codec': 'unknown', 'duration_seconds': None, } -@pytest.mark.parametrize( - ('summary', 'expected'), - [ - # Happy path: H.264 + AAC in mp4 - ( - {'container': 'mp4', 'video_codec': 'h264', 'audio_codec': 'aac'}, - True, - ), - # HEVC in mkv with no audio (board profile must allow hevc) - ( - {'container': 'mkv', 'video_codec': 'hevc', 'audio_codec': 'none'}, - True, - ), - # Unknown container — fail - ( - {'container': 'avs', 'video_codec': 'h264', 'audio_codec': 'aac'}, - False, - ), - # Exotic codec — fail - ( - { - 'container': 'mov', - 'video_codec': 'prores', - 'audio_codec': 'pcm_s16le', - }, - False, - ), - # Unknown audio codec — fail (we'd have to demux it out) - ( - { - 'container': 'mp4', - 'video_codec': 'h264', - 'audio_codec': 'truehd', - }, - False, - ), - # All unknowns (probe failed) — fail safely → transcode - ( - { - 'container': 'unknown', - 'video_codec': 'unknown', - 'audio_codec': 'unknown', - }, - False, - ), - ], -) -def test_video_can_passthrough_decision_table( - summary: dict[str, str], expected: bool -) -> None: - """Exhaustive truth table for ``_video_can_passthrough``. Catches - a future change to the passthrough sets that wasn't intended. - Pins the board profile to ``pi5`` (which accepts both h264 + hevc) - so the legacy "happy path" cases stay equivalent — separate - per-board tests below cover the pi2/pi3 H.264-only branch.""" - pi5_profile = processing._BOARD_PROFILES['pi5'] - assert processing._video_can_passthrough(summary, pi5_profile) is expected - - -# --------------------------------------------------------------------------- -# Per-board transcode profile (the codec grid) -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize( - ('device_type', 'expected_target'), - [ - ('pi2', 'h264'), - ('pi3', 'h264'), - ('pi4-64', 'hevc'), - ('pi5', 'hevc'), - ('x86', 'hevc'), - # Unset / unknown env var falls back to H.264 — the most - # compatible codec for any Anthias-supported device. - ('', 'h264'), - ('weird-future-board', 'h264'), - ], -) -def test_resolve_board_profile_picks_target_codec_per_board( - device_type: str, expected_target: str, monkeypatch: pytest.MonkeyPatch -) -> None: - """The transcode target lives in a board profile keyed by - ``DEVICE_TYPE``. This regression-tests the grid in one place so a - future "let's also build a pi6 image" rollout can't silently fall - through to H.264 if it forgets to register a profile entry.""" - monkeypatch.setenv('DEVICE_TYPE', device_type) - profile = processing._resolve_board_profile() - assert profile['transcode_target'] == expected_target - - -@pytest.mark.parametrize( - ('device_type', 'video_codec', 'expected_passthrough'), - [ - # pi2 / pi3: VLC + mmal-vc4. H.264 only. HEVC must transcode. - ('pi2', 'h264', True), - ('pi2', 'hevc', False), - ('pi3', 'h264', True), - ('pi3', 'hevc', False), - # pi4-64 / pi5 / x86: mpv with HEVC support. Both codecs OK. - ('pi4-64', 'h264', True), - ('pi4-64', 'hevc', True), - ('pi5', 'h264', True), - ('pi5', 'hevc', True), - ('x86', 'h264', True), - ('x86', 'hevc', True), - # Default profile is H.264-only — safer for unknown boards. - ('', 'h264', True), - ('', 'hevc', False), - ], -) -def test_video_can_passthrough_respects_board_codec_set( - device_type: str, - video_codec: str, - expected_passthrough: bool, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """A pi3 device must not passthrough an HEVC upload; a pi5 device - must. The test pins ``DEVICE_TYPE`` rather than passing the - profile explicitly so the env-resolution code path is exercised - end-to-end (mirrors how the celery worker decides at runtime).""" - monkeypatch.setenv('DEVICE_TYPE', device_type) - summary = { - 'container': 'mp4', - 'video_codec': video_codec, - 'audio_codec': 'aac', - } - assert processing._video_can_passthrough(summary) is expected_passthrough - - -@pytest.mark.parametrize( - ('device_type', 'expected_codec', 'expected_extra'), - [ - ('pi2', 'libx264', None), - ('pi3', 'libx264', None), - ('pi4-64', 'libx265', 'hvc1'), - ('pi5', 'libx265', 'hvc1'), - ('x86', 'libx265', 'hvc1'), - ('', 'libx264', None), - ], -) -def test_transcode_to_target_uses_board_specific_encoder( - device_type: str, - expected_codec: str, - expected_extra: str | None, - monkeypatch: pytest.MonkeyPatch, -) -> None: - """Capture the ffmpeg argv ``_transcode_to_target`` invokes and - assert the encoder + ``-tag:v hvc1`` (HEVC only) match the - board's expected output. Mocks ``sh.ffmpeg`` so no actual encode - runs — we only care about the argv shape here.""" - monkeypatch.setenv('DEVICE_TYPE', device_type) - - captured: dict[str, Any] = {} - - def fake_ffmpeg(*args: Any, **kwargs: Any) -> None: - captured['args'] = list(args) - captured['kwargs'] = kwargs - - with mock.patch.object(sh, 'ffmpeg', side_effect=fake_ffmpeg): - processing._transcode_to_target('in.mov', 'out.mp4') - - args = captured['args'] - # ``-c:v `` lands somewhere in the middle of the argv. - assert '-c:v' in args - codec_index = args.index('-c:v') - assert args[codec_index + 1] == expected_codec - # AAC audio + faststart are invariants across boards. - assert '-c:a' in args and 'aac' in args - assert '-movflags' in args and '+faststart' in args - assert '-threads' in args and '2' in args - if expected_extra == 'hvc1': - # HEVC output gets the iOS-friendly hvc1 codec tag. - assert '-tag:v' in args - tag_index = args.index('-tag:v') - assert args[tag_index + 1] == 'hvc1' - else: - assert '-tag:v' not in args - - -@pytest_ffmpeg -@pytest.mark.django_db -def test_video_passthrough_records_target_codec( - asset_dir: str, monkeypatch: pytest.MonkeyPatch -) -> None: - """Passthrough rows still get ``transcode_target`` written so the - operator can see "this device wanted hevc, the upload already was - hevc, no work needed".""" - monkeypatch.setenv('DEVICE_TYPE', 'pi5') - src = path.join(asset_dir, 'sample.mp4') - _make_video(src, codec='libx264', container='mp4', audio='aac') - asset = _make_processing_asset('vid-pass-pi5', src, mimetype='video') - - with mock.patch.object(processing, '_notify'): - processing._run_video_normalisation(asset) - - asset.refresh_from_db() - assert asset.metadata['transcoded'] is False - assert asset.metadata['transcode_target'] == 'hevc' - - -@pytest.mark.django_db -def test_video_pi3_transcodes_hevc_to_h264( - asset_dir: str, monkeypatch: pytest.MonkeyPatch -) -> None: - """A pi3 device receiving an HEVC upload must transcode to H.264 - even though the source is in an accepted container — pi3's VLC + - mmal-vc4 path can't decode HEVC. Mocks the actual ffmpeg run so - the test doesn't depend on libx265 being available; asserts on - the captured argv to lock in the codec choice.""" - monkeypatch.setenv('DEVICE_TYPE', 'pi3') - src = path.join(asset_dir, 'fixture.mkv') - with open(src, 'wb') as fh: - fh.write(b'\x00' * 64) - asset = _make_processing_asset('vid-pi3-hevc', src, mimetype='video') - - summary = { - 'container': 'mkv', - 'video_codec': 'hevc', - 'audio_codec': 'aac', - } - captured: dict[str, Any] = {} - - def fake_transcode(_in: str, staging: str, _profile: Any = None) -> None: - # Capture the profile that was selected and produce a stub - # output so the runner can finalise the row. - captured['profile'] = _profile - with open(staging, 'wb') as fh: - fh.write(b'\x00\x00\x00\x18ftypmp42') - - with ( - mock.patch.object(processing, '_notify'), - mock.patch.object( - processing, '_ffprobe_summary', return_value=summary - ), - mock.patch.object( - processing, '_transcode_to_target', side_effect=fake_transcode - ), - mock.patch.object( - processing, '_resolve_duration_seconds', return_value=10 - ), - ): - processing._run_video_normalisation(asset) - - asset.refresh_from_db() - assert asset.metadata['transcoded'] is True - assert asset.metadata['transcode_target'] == 'h264' - # The runner threaded the resolved profile to the transcode - # helper rather than letting it re-resolve from env (which would - # also be correct, but threading is the cheaper invariant). - assert captured['profile']['transcode_target'] == 'h264' - - # --------------------------------------------------------------------------- # Celery wrapper tests — task-level behaviour (no_op guards, on_failure) # --------------------------------------------------------------------------- @@ -1741,6 +1324,75 @@ def test_normalize_on_failure_no_args_is_safe() -> None: task.on_failure(RuntimeError('boom'), 'task-id', (), {}, None) +@pytest.mark.django_db +def test_normalize_on_failure_unsupported_codec_persists_recipe( + asset_dir: str, +) -> None: + """``UnsupportedVideoCodecError`` is the gate's user-facing + exception. on_failure must: + + * write the bare message into ``metadata.error_message`` (no + ``UnsupportedVideoCodecError:`` class-name prefix — that's the + P1 review finding), and + * mirror the exception's ``recipe`` attribute into + ``metadata.error_recipe`` so the Edit modal can render it in a + copyable ```` block. + """ + asset = _make_processing_asset( + 'vid-onfail', + path.join(asset_dir, 'fixture.mpg'), + mimetype='video', + ) + task = processing._NormalizeAssetTask() + exc = processing.UnsupportedVideoCodecError( + "Video codec 'mpeg2video' is not hardware-decoded on this " + 'device. Supported: h264, hevc.', + recipe="ffmpeg -i 'fixture.mpg' -c:v libx264 'fixture.mp4'", + ) + + with mock.patch.object(processing, '_notify'): + task.on_failure(exc, 'task-id', (asset.asset_id,), {}, None) + + asset.refresh_from_db() + assert asset.is_processing is False + msg = asset.metadata['error_message'] + assert 'mpeg2video' in msg + assert 'UnsupportedVideoCodecError' not in msg + assert asset.metadata['error_recipe'] == ( + "ffmpeg -i 'fixture.mpg' -c:v libx264 'fixture.mp4'" + ) + + +@pytest.mark.django_db +def test_normalize_on_failure_clears_stale_error_recipe( + asset_dir: str, +) -> None: + """A subsequent non-recipe failure must clear any stale + ``error_recipe`` from a previous run, otherwise the modal would + show an outdated recipe alongside the new error message.""" + asset = _make_processing_asset( + 'img-clears', + path.join(asset_dir, 'fixture.tiff'), + mimetype='image', + metadata={ + 'error_recipe': "ffmpeg -i 'old.mpg' -c:v libx264 'old.mp4'", + }, + ) + task = processing._NormalizeAssetTask() + + with mock.patch.object(processing, '_notify'): + task.on_failure( + UnidentifiedImageError('cannot decode'), + 'task-id', + (asset.asset_id,), + {}, + None, + ) + + asset.refresh_from_db() + assert 'error_recipe' not in asset.metadata + + # --------------------------------------------------------------------------- # Helper / dispatch tests # --------------------------------------------------------------------------- @@ -1991,43 +1643,6 @@ def __str__(self) -> str: assert result['streams'][0]['codec_name'] == 'h264' -@pytest.mark.django_db -def test_video_passthrough_skips_duration_when_probe_unavailable( - asset_dir: str, -) -> None: - """If ffprobe is unavailable (host without ffmpeg apt package), - the passthrough branch still flips is_processing — the row - stays at its placeholder duration so the operator can edit it - manually rather than being stuck.""" - src = path.join(asset_dir, 'fixture.mp4') - with open(src, 'wb') as fh: - # Just enough so isfile() passes; the probe is mocked anyway. - fh.write(b'\x00' * 64) - asset = _make_processing_asset('vid-noprobe', src, mimetype='video') - - summary = { - 'container': 'mp4', - 'video_codec': 'h264', - 'audio_codec': 'aac', - } - with ( - mock.patch.object( - processing, '_ffprobe_summary', return_value=summary - ), - mock.patch.object( - processing, '_resolve_duration_seconds', return_value=None - ), - mock.patch.object(processing, '_notify'), - ): - processing._run_video_normalisation(asset) - - asset.refresh_from_db() - assert asset.is_processing is False - # Duration left at the placeholder — never overwritten with None. - assert asset.duration == 0 - assert asset.metadata['transcoded'] is False - - # --------------------------------------------------------------------------- # Static webp fixture for upload-path tests # --------------------------------------------------------------------------- diff --git a/tools/image_builder/utils.py b/tools/image_builder/utils.py index c4d6d2554..b67596e39 100644 --- a/tools/image_builder/utils.py +++ b/tools/image_builder/utils.py @@ -181,15 +181,20 @@ def get_viewer_context(board: str, target_platform: str) -> dict[str, Any]: # from apt at runtime — except on Qt 6 boards where qt6-*-dev # below also provides the runtime libs.) # - # X11/XCB packages are intentionally absent: the WebView is - # configured with `-no-xcb -no-xcb-xlib -qpa eglfs` (see - # src/anthias_webview/build_qt5.sh) and runs under - # QT_QPA_PLATFORM=linuxfb - # straight on KMS/DRM, so Qt has no X code path to dlopen. mpv - # uses --vo=drm. Wayland is similarly absent on Pi for the same - # reason; the x86 board is the one exception (it has no /dev/fb0, - # so the qt6-wayland + cage pair is added to the per-board apt - # extension below). + # X11/XCB packages are intentionally absent. Two display tracks + # for the four image targets (no X code path on either): + # + # * Pi2 / Pi3 / Pi4-64: Qt linuxfb + mpv straight to KMS. Pi 2 / + # Pi 3 use a custom -no-xcb -no-xcb-xlib -qpa eglfs Qt 5 + # WebView build (see src/anthias_webview/build_qt5.sh) with mpv + # --vo=drm. Pi 4 is Qt 6 with the same + # QT_QPA_PLATFORM=linuxfb plus mpv --vo=gpu --gpu-context=drm — + # V3D-accelerated scaling without the cage composite pass the + # V3D 6.0 can't keep up with. + # * Pi5 / x86 / arm64: cage (a kiosk wlroots compositor) with + # QT_QPA_PLATFORM=wayland and mpv --vo=gpu + # --gpu-context=wayland. The cage + qt6-wayland + wlr-randr + # triple is added to the per-board apt extension below. viewer_extra_apt_dependencies = [ 'ca-certificates', 'dbus-daemon', @@ -253,33 +258,42 @@ def get_viewer_context(board: str, target_platform: str) -> dict[str, Any]: ] if is_qt6: - # pi4-64/pi5 use mpv --vo=drm; x86 uses mpv --vo=gpu - # --gpu-context=wayland under cage with VAAPI hwdec (see - # MPVMediaPlayer.play in src/anthias_viewer/media_player.py). - # VLC is deliberately *not* installed: MediaPlayerProxy routes - # Qt6 boards to MPVMediaPlayer, so VLC would just be ~80–100 MB - # of dead weight here. + # Shared Qt 6 runtime for every Qt6 board (pi4-64, pi5, x86, + # arm64). mpv handles video for all of them via MPVMediaPlayer; + # VLC is deliberately *not* installed because MediaPlayerProxy + # never routes Qt6 boards to it (would be ~80-100 MB of dead + # weight). viewer_extra_apt_dependencies.extend( [ 'mpv', 'qt6-base-dev', - 'qt6-webengine-dev', 'qt6-image-formats-plugins', + 'qt6-webengine-dev', ] ) - if board in ('x86', 'arm64'): - # balenaOS x86 has no /dev/fb0 for Qt's linuxfb plugin and - # no host display server. cage is a kiosk wlroots - # compositor that talks straight to KMS; qt6-wayland is - # the Qt platform plugin the viewer loads to render into - # cage's surface. The same wiring fits arm64 - # (non-Pi 64-bit ARM SBCs running Armbian): no /dev/fb0 - # by default, so cage is the portable kiosk path. + if board in ('x86', 'arm64', 'pi5'): + # cage is a kiosk wlroots compositor that talks straight + # to KMS; qt6-wayland is the Qt platform plugin the + # viewer loads to render into cage's surface; mpv talks + # to the same Wayland socket via --vo=gpu + # --gpu-context=wayland (see MPVMediaPlayer.play in + # src/anthias_viewer/media_player.py). wlr-randr is how + # src/anthias_viewer/__init__.py applies the Settings + # page's "screen rotation" knob — Qt's wayland QPA has + # no rotation= equivalent, so the transform goes through + # the compositor for both Qt and mpv consistently. + # + # Pi 4 is intentionally NOT on this path: the V3D 6.0 + # doesn't have the bandwidth to composite cage on top of + # video. It stays on Qt linuxfb + mpv --vo=gpu + # --gpu-context=drm — see bin/start_viewer.sh and + # docker/Dockerfile.viewer.j2. viewer_extra_apt_dependencies.extend( [ 'cage', 'qt6-wayland', + 'wlr-randr', ] ) @@ -292,22 +306,16 @@ def get_viewer_context(board: str, target_platform: str) -> dict[str, Any]: # picks whichever VAAPI driver matches the device at # runtime. # - # Deliberately NOT shipped on arm64: Rockchip - # (rkvdec/hantro), Allwinner (cedrus), and Amlogic - # (meson-vdec) all expose hardware decode via V4L2 M2M / - # request API, not VAAPI; mesa-va-drivers only covers - # radeonsi/nouveau/etc., so on those SoCs va-driver-all - # would just be dead weight. Per-SoC hwdec for ARM SBCs - # is a Tier-2 follow-up. + # Deliberately NOT shipped on arm64/Pi: Rockchip + # (rkvdec/hantro), Allwinner (cedrus), Amlogic + # (meson-vdec), and the Pi V3D all expose hardware decode + # via V4L2 M2M / request API, not VAAPI; mesa-va-drivers + # only covers radeonsi/nouveau/etc., so on those SoCs + # va-driver-all would just be dead weight. Per-SoC hwdec + # for those boards is a Tier-2 follow-up. viewer_extra_apt_dependencies.extend( [ 'va-driver-all', - # wlr-randr is how the viewer applies the Settings - # page's "screen rotation" knob on x86 — Qt's - # wayland QPA has no rotation= equivalent, so the - # transform has to go through the compositor. - # src/anthias_viewer/__init__.py drives this. - 'wlr-randr', ] ) else: diff --git a/uv.lock b/uv.lock index 4726c5cc3..dda5ba7d5 100644 --- a/uv.lock +++ b/uv.lock @@ -45,7 +45,7 @@ wheels = [ [[package]] name = "ansible-core" -version = "2.19.9" +version = "2.20.5" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "cryptography" }, @@ -54,9 +54,9 @@ dependencies = [ { name = "pyyaml" }, { name = "resolvelib" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/26/5b/161275ff5fa673dfc1329ed19a44331a4f491166a9f0cb907b44037ad72e/ansible_core-2.19.9.tar.gz", hash = "sha256:74107de13d188ff579fb215bc3eb875c9198803215d6378ed588c7f35aba12f5", size = 3431475, upload-time = "2026-04-21T00:47:09.194Z" } +sdist = { url = "https://files.pythonhosted.org/packages/9d/ec/690cc73e38c3546eabc8ef4118e0d7be1758a598bc23eed3e24ca1f346a7/ansible_core-2.20.5.tar.gz", hash = "sha256:82e3049d95e6e02e5d20d4a5a8e10533a55e0cc52e878e4cf77166c45410f16f", size = 3339511, upload-time = "2026-04-21T00:48:27.175Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/3d/b9/f2c07e6cdba6a5882342d666ceea4eeeb6fea20ec325fac68c27bfcea722/ansible_core-2.19.9-py3-none-any.whl", hash = "sha256:18de80e3ea5a89d2ea84e4d4a3c6d7c121ea62fc3af2877ba083784e3d90b419", size = 2416269, upload-time = "2026-04-21T00:47:06.777Z" }, + { url = "https://files.pythonhosted.org/packages/f9/e1/4454505e725b84ae670229565dfc20c4075480199647bf4874cf337c560e/ansible_core-2.20.5-py3-none-any.whl", hash = "sha256:ff6ff15c6a37fda07dc7400207e17e93727b24173ca48c068b3311a50d75ecc3", size = 2416843, upload-time = "2026-04-21T00:48:25.413Z" }, ] [[package]] @@ -281,7 +281,7 @@ dev-host = [ { name = "types-gunicorn", specifier = "==25.3.0.20260408" }, { name = "types-netifaces", specifier = "==0.11.0.20260408" }, { name = "types-python-dateutil", specifier = "==2.9.0.20260408" }, - { name = "types-pytz", specifier = "==2026.1.1.20260408" }, + { name = "types-pytz", specifier = "==2026.2.0.20260506" }, { name = "types-pyyaml", specifier = "==6.0.12.20260408" }, { name = "types-requests", specifier = "==2.33.0.20260408" }, ] @@ -293,7 +293,7 @@ docker-image-builder = [ { name = "requests", specifier = "==2.33.1" }, ] host = [ - { name = "ansible-core", specifier = "==2.19.9" }, + { name = "ansible-core", specifier = "==2.20.5" }, { name = "netifaces2", specifier = "==0.0.22" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", specifier = "==2.33.1" }, @@ -314,7 +314,7 @@ mypy = [ { name = "django-dbbackup", specifier = "==5.3.0" }, { name = "django-stubs", specifier = "==6.0.3" }, { name = "django-stubs-ext", specifier = "==6.0.3" }, - { name = "djangorestframework", specifier = "==3.16.1" }, + { name = "djangorestframework", specifier = "==3.17.1" }, { name = "djangorestframework-stubs", specifier = "==3.16.9" }, { name = "drf-spectacular", specifier = "==0.29.0" }, { name = "jinja2", specifier = "==3.1.6" }, @@ -322,14 +322,14 @@ mypy = [ { name = "pillow", specifier = "==12.2.0" }, { name = "pygit2", specifier = "==1.19.1" }, { name = "python-on-whales", specifier = "==0.81.0" }, - { name = "pytz", specifier = "==2025.2" }, + { name = "pytz", specifier = "==2026.2" }, { name = "requests", specifier = "==2.33.1" }, { name = "ruff", specifier = "==0.14.10" }, { name = "tenacity", specifier = "==9.1.4" }, { name = "types-gunicorn", specifier = "==25.3.0.20260408" }, { name = "types-netifaces", specifier = "==0.11.0.20260408" }, { name = "types-python-dateutil", specifier = "==2.9.0.20260408" }, - { name = "types-pytz", specifier = "==2026.1.1.20260408" }, + { name = "types-pytz", specifier = "==2026.2.0.20260506" }, { name = "types-pyyaml", specifier = "==6.0.12.20260408" }, { name = "types-requests", specifier = "==2.33.0.20260408" }, { name = "whitenoise", specifier = "==6.12.0" }, @@ -343,17 +343,17 @@ server = [ { name = "django", specifier = "==5.2.14" }, { name = "django-dbbackup", specifier = "==5.3.0" }, { name = "django-stubs-ext", specifier = "==6.0.3" }, - { name = "djangorestframework", specifier = "==3.16.1" }, + { name = "djangorestframework", specifier = "==3.17.1" }, { name = "drf-spectacular", specifier = "==0.29.0" }, { name = "jinja2", specifier = "==3.1.6" }, { name = "netifaces", specifier = "==0.11.0" }, { name = "packaging", specifier = "==26.1" }, { name = "pillow", specifier = "==12.2.0" }, - { name = "pillow-heif", specifier = "==1.2.1" }, + { name = "pillow-heif", specifier = "==1.3.0" }, { name = "psutil", specifier = "==7.2.1" }, { name = "pydbus", specifier = "==0.6.0" }, { name = "python-dateutil", specifier = "==2.9.0.post0" }, - { name = "pytz", specifier = "==2025.2" }, + { name = "pytz", specifier = "==2026.2" }, { name = "pyyaml", specifier = "==6.0.2" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", specifier = "==2.33.1" }, @@ -373,13 +373,13 @@ test = [ { name = "django", specifier = "==5.2.14" }, { name = "django-dbbackup", specifier = "==5.3.0" }, { name = "django-stubs-ext", specifier = "==6.0.3" }, - { name = "djangorestframework", specifier = "==3.16.1" }, + { name = "djangorestframework", specifier = "==3.17.1" }, { name = "drf-spectacular", specifier = "==0.29.0" }, { name = "jinja2", specifier = "==3.1.6" }, { name = "netifaces", specifier = "==0.11.0" }, { name = "packaging", specifier = "==26.1" }, { name = "pillow", specifier = "==12.2.0" }, - { name = "pillow-heif", specifier = "==1.2.1" }, + { name = "pillow-heif", specifier = "==1.3.0" }, { name = "playwright", specifier = "==1.59.0" }, { name = "psutil", specifier = "==7.2.1" }, { name = "pydbus", specifier = "==0.6.0" }, @@ -391,7 +391,7 @@ test = [ { name = "pytest-xdist", specifier = "==3.6.1" }, { name = "python-dateutil", specifier = "==2.9.0.post0" }, { name = "python-vlc", specifier = "==3.0.21203" }, - { name = "pytz", specifier = "==2025.2" }, + { name = "pytz", specifier = "==2026.2" }, { name = "pyyaml", specifier = "==6.0.2" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", specifier = "==2.33.1" }, @@ -412,7 +412,7 @@ viewer = [ { name = "pydbus", specifier = "==0.6.0" }, { name = "python-dateutil", specifier = "==2.9.0.post0" }, { name = "python-vlc", specifier = "==3.0.21203" }, - { name = "pytz", specifier = "==2025.2" }, + { name = "pytz", specifier = "==2026.2" }, { name = "redis", specifier = "==7.4.0" }, { name = "requests", specifier = "==2.33.1" }, { name = "sh", specifier = "==2.2.2" }, @@ -905,14 +905,14 @@ wheels = [ [[package]] name = "djangorestframework" -version = "3.16.1" +version = "3.17.1" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "django" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/8a/95/5376fe618646fde6899b3cdc85fd959716bb67542e273a76a80d9f326f27/djangorestframework-3.16.1.tar.gz", hash = "sha256:166809528b1aced0a17dc66c24492af18049f2c9420dbd0be29422029cfc3ff7", size = 1089735, upload-time = "2025-08-06T17:50:53.251Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ca/d7/c016e69fac19ff8afdc89db9d31d9ae43ae031e4d1993b20aca179b8301a/djangorestframework-3.17.1.tar.gz", hash = "sha256:a6def5f447fe78ff853bff1d47a3c59bf38f5434b031780b351b0c73a62db1a5", size = 905742, upload-time = "2026-03-24T16:58:33.705Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/b0/ce/bf8b9d3f415be4ac5588545b5fcdbbb841977db1c1d923f7568eeabe1689/djangorestframework-3.16.1-py3-none-any.whl", hash = "sha256:33a59f47fb9c85ede792cbf88bde71893bcda0667bc573f784649521f1102cec", size = 1080442, upload-time = "2025-08-06T17:50:50.667Z" }, + { url = "https://files.pythonhosted.org/packages/5a/e1/2c516bdc83652b1a60c6119366ac2c0607b479ed05cd6093f916ca8928f8/djangorestframework-3.17.1-py3-none-any.whl", hash = "sha256:c3c74dd3e83a5a3efc37b3c18d92bd6f86a6791c7b7d4dff62bb068500e76457", size = 898844, upload-time = "2026-03-24T16:58:31.845Z" }, ] [[package]] @@ -1340,27 +1340,34 @@ wheels = [ [[package]] name = "pillow-heif" -version = "1.2.1" +version = "1.3.0" source = { registry = "https://pypi.org/simple" } dependencies = [ { name = "pillow" }, ] -sdist = { url = "https://files.pythonhosted.org/packages/22/f4/68bd0465dc0494c22e23334dde0a9c52dec5afe98cf5a40abb47f75e1b08/pillow_heif-1.2.1.tar.gz", hash = "sha256:29be44d636269e2d779b4aec629bc056ec7260b734a16b4d3bb284c49c200274", size = 17128668, upload-time = "2026-02-18T11:20:48.643Z" } -wheels = [ - { url = "https://files.pythonhosted.org/packages/11/79/58d3a9cce7025e648588e02d71a73f952d97a41ec38d3462f72e67693998/pillow_heif-1.2.1-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:246acfade36d877fc7e01ccde03edaafd75e5aad66f889f484fc8ba7b651b688", size = 4666937, upload-time = "2026-02-18T11:20:17.09Z" }, - { url = "https://files.pythonhosted.org/packages/6e/0e/82afa841a728e5d143df00b674d93cb7768ea28ecb2620c5360696298a5c/pillow_heif-1.2.1-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:4a746d38f2c54774fd680da45f2af56467b15f6b6c46962328ad1ed005d16ca6", size = 3392298, upload-time = "2026-02-18T11:20:19.014Z" }, - { url = "https://files.pythonhosted.org/packages/35/05/72cf44450aa9766f15d5aa22d311cc371d9dcdc9a08569cdffe98e47003a/pillow_heif-1.2.1-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:a93e374ff86ef61dc374a6d3c22e73fddc609e10b342802fa1674cc26db50859", size = 5843470, upload-time = "2026-02-18T11:20:20.585Z" }, - { url = "https://files.pythonhosted.org/packages/6e/89/f00a3a9c379c6730fdd79244ce44e8e6262f523554b2055426cf430d4459/pillow_heif-1.2.1-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:f8f0158a0368a38870deda5124d74086f8708268f335ddbdeb0890ef83ecd7ad", size = 5577875, upload-time = "2026-02-18T11:20:22.034Z" }, - { url = "https://files.pythonhosted.org/packages/81/64/c370288ac526c9ef846595490e115dcd7ff706e6c288c98b4eb7794e5ec9/pillow_heif-1.2.1-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:5132c9c84e18ca800d559b79e389114b289899614c241e4399f8b677f1bbd3d7", size = 6885247, upload-time = "2026-02-18T11:20:23.625Z" }, - { url = "https://files.pythonhosted.org/packages/68/cf/bc43a3e1f5284e8c579a6bd637be0e4a28e8e9b633625d2f6406513dcf31/pillow_heif-1.2.1-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:c31012a51fe3d67ee0c6c91549a5ee0590f3fa07b03882022238d0d0f052ad20", size = 6510200, upload-time = "2026-02-18T11:20:25.33Z" }, - { url = "https://files.pythonhosted.org/packages/55/94/8165b5c024ee84617af4c4888d961ec642de627c9329e9c6312e23453dab/pillow_heif-1.2.1-cp313-cp313-win_amd64.whl", hash = "sha256:e27d7690a08f52c63295f5ca5e13b97bbe168f2f55e32794e3b24898a5270255", size = 5483260, upload-time = "2026-02-18T11:20:26.83Z" }, - { url = "https://files.pythonhosted.org/packages/9f/f3/0c96dc4c9cdcb6c0e9c2fbb75c924e205716283e8aec1da63eac279f7f3c/pillow_heif-1.2.1-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:a9083f80271130580e6f99f6b79204fc7f5ff61eefb83ac64c026c68f0000775", size = 4666976, upload-time = "2026-02-18T11:20:28.138Z" }, - { url = "https://files.pythonhosted.org/packages/47/71/935482ccbbc8cbf94b945eb2f6e18718e455ad5d35b4356ca0233a273790/pillow_heif-1.2.1-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:b3584abf861d33a422a7bda1f2926131cbf4bbd2801390cb7f75f03ef3833a2d", size = 3392384, upload-time = "2026-02-18T11:20:29.418Z" }, - { url = "https://files.pythonhosted.org/packages/81/b5/260bd9f6043ffe286e5afcc6c83b71336b98ec55a01f75d665479a053fac/pillow_heif-1.2.1-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:be790ce430645c3e0b148e873ed5ebeeb6d001ae685e8db40f77f43474ab9848", size = 5843664, upload-time = "2026-02-18T11:20:30.716Z" }, - { url = "https://files.pythonhosted.org/packages/f6/37/1309eca6039bdf1120711eaf722ddb03c46152316e0c53ce888467ff5931/pillow_heif-1.2.1-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:957060f8f2ceaa2e1fd41450da05bee87abc054a6247c02b53e9322ce4e53958", size = 5577989, upload-time = "2026-02-18T11:20:32.758Z" }, - { url = "https://files.pythonhosted.org/packages/7e/ae/17ba507ed172fefd0d89e57e68c96f0e31213de240624a24221ed7c9a1ab/pillow_heif-1.2.1-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:fe231ca4c4e387785a97f2acf38a24474f3a0819b7e2234144cff9fa3de5d3ac", size = 6885375, upload-time = "2026-02-18T11:20:34.37Z" }, - { url = "https://files.pythonhosted.org/packages/19/c3/39109edf56c039793f15089bda0254cbfcf57f884bc82fe953ae36c36b66/pillow_heif-1.2.1-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:f780890596161c7f43512377dda9106f793421565a376c70988355de5c4241de", size = 6510323, upload-time = "2026-02-18T11:20:36.369Z" }, - { url = "https://files.pythonhosted.org/packages/e9/24/e3e3ed81aaeaa9c6dcb753f796ca7576426e9216f0973edd9233b3c223e1/pillow_heif-1.2.1-cp314-cp314-win_amd64.whl", hash = "sha256:eee8c933cce88dc8f6a01afc3befc159341fbc404a981c3759b3dca97b7f2dbb", size = 5640667, upload-time = "2026-02-18T11:20:38.252Z" }, +sdist = { url = "https://files.pythonhosted.org/packages/cd/58/2df4fc42840633e01c97b75965cb1bc6e14425973b92382391650e97e4b7/pillow_heif-1.3.0.tar.gz", hash = "sha256:af8d2bda85e395677d5bb50d7bda3b5655c946cc95b913b5e7222fabacbb467f", size = 17133211, upload-time = "2026-02-27T12:21:36.465Z" } +wheels = [ + { url = "https://files.pythonhosted.org/packages/81/c3/9effa6ab5c2c2ffb80228143c578a9a2a8e2f059dd9d067ec6ff6f6c89db/pillow_heif-1.3.0-cp313-cp313-macosx_10_15_x86_64.whl", hash = "sha256:641c50a064aa9ad6626a6b2b914b65855202f937d573d53838e344feb2e8c6d1", size = 4667379, upload-time = "2026-02-27T12:20:57.561Z" }, + { url = "https://files.pythonhosted.org/packages/23/eb/b6b52e3655f366b95301f18aecd2d35487cace18d17134b80ad0f70cc1eb/pillow_heif-1.3.0-cp313-cp313-macosx_11_0_arm64.whl", hash = "sha256:9390dd7987887aa09779fbd88bbab715c732c9ad3a71d6707284035e3ca93379", size = 3392725, upload-time = "2026-02-27T12:20:59.52Z" }, + { url = "https://files.pythonhosted.org/packages/c1/b3/b69610e9565fc8bcaf2303f412e857c0439d23cc18cf866c72a96ec6b2e6/pillow_heif-1.3.0-cp313-cp313-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:6e8444ccb330015e1db930207d269886e4b6c666121cd9e5fdad88735950b09f", size = 5844285, upload-time = "2026-02-27T12:21:00.771Z" }, + { url = "https://files.pythonhosted.org/packages/47/8c/be44f6dea425a9756ff418cb03f5ee75ed1c7dd1ff9bee1f3893b2b82da4/pillow_heif-1.3.0-cp313-cp313-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:7d30054ccc97ecbe5ee3fa486a505ccc33bfbb27f005ad624ddb4c17b80ddd57", size = 5578691, upload-time = "2026-02-27T12:21:02.193Z" }, + { url = "https://files.pythonhosted.org/packages/5c/74/e12d49346a39e2204b408a835b31b2fd9a5d51f97ce3a6015cf22ca09a54/pillow_heif-1.3.0-cp313-cp313-musllinux_1_2_aarch64.whl", hash = "sha256:dc1b9c9efdf8345d703118449ff69696d0827bdf28e3b52f82015f5714f7c23e", size = 6885923, upload-time = "2026-02-27T12:21:03.782Z" }, + { url = "https://files.pythonhosted.org/packages/80/a6/51c937a9433f5ae9c625b686ee338bdf0080a1661f7eb34daaf75424ee77/pillow_heif-1.3.0-cp313-cp313-musllinux_1_2_x86_64.whl", hash = "sha256:ee26b2155721e7f5f7b10fa93ca2ad3be59547c5c5e5d9d50e6ea17531b81d60", size = 6511216, upload-time = "2026-02-27T12:21:05.134Z" }, + { url = "https://files.pythonhosted.org/packages/63/0a/bb8435e127f75b434166022471bbabf11c8c1fc3d48c8595fd6ab36c2785/pillow_heif-1.3.0-cp313-cp313-win_amd64.whl", hash = "sha256:17ecbbadfe10ea12a65c1c12354dc1ed8ae1e5d1b7092ea753641b029f7d6f9e", size = 5483570, upload-time = "2026-02-27T12:21:06.566Z" }, + { url = "https://files.pythonhosted.org/packages/3e/17/aa056f8edb71396dd1131abcd0c6feab00097ceec89a12fc62d2dbc3ccf5/pillow_heif-1.3.0-cp314-cp314-macosx_10_15_x86_64.whl", hash = "sha256:8267a73d3b2d07a47a96428bd8cd4c406e1637a94f29d4c16ce08b31b8e50a07", size = 4667395, upload-time = "2026-02-27T12:21:08.16Z" }, + { url = "https://files.pythonhosted.org/packages/19/1f/da50ccd271a2878d17df359301dc2f7a79ec1cbb6e92c19ccc8c6219d497/pillow_heif-1.3.0-cp314-cp314-macosx_11_0_arm64.whl", hash = "sha256:36bbea7679467caa3a154db11c04f1ca2fa8591e886f06f40f7831c14b58d771", size = 3392800, upload-time = "2026-02-27T12:21:09.668Z" }, + { url = "https://files.pythonhosted.org/packages/11/bc/1f89d927c1293cf283bc5d0ae6735d268d2de9749aa6fb94342ec838a457/pillow_heif-1.3.0-cp314-cp314-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:0ea3a4b2de4b6c63407af72afdac901616807c6e6a030fe77851d227bca3727a", size = 5844547, upload-time = "2026-02-27T12:21:10.826Z" }, + { url = "https://files.pythonhosted.org/packages/0f/04/d781b23f8bff125c8dd8da63d928a35e38f2b727e89582a1fd323664e968/pillow_heif-1.3.0-cp314-cp314-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:05149bd26b08dae5af7a389af6db13cef4f12c7871db73d84e40a1f3c83b0142", size = 5578827, upload-time = "2026-02-27T12:21:12.06Z" }, + { url = "https://files.pythonhosted.org/packages/2a/98/8dcdaafcf9bd8b26ed0569dc93653dc20a06faef7bfbdd4ba05c091c5b60/pillow_heif-1.3.0-cp314-cp314-musllinux_1_2_aarch64.whl", hash = "sha256:f8b7a50058fc3152f42b68aa2b30601249f61aa5c6c27876af076785c7051fd9", size = 6886088, upload-time = "2026-02-27T12:21:13.635Z" }, + { url = "https://files.pythonhosted.org/packages/99/26/93f3c8bfffb7e8fe0244bf86117235c49c23980e61320e7484c03ac836e2/pillow_heif-1.3.0-cp314-cp314-musllinux_1_2_x86_64.whl", hash = "sha256:edb3ef437e8841475db14721f0529e600bb55c41b549ad1794a0831e28f33bac", size = 6511291, upload-time = "2026-02-27T12:21:15.354Z" }, + { url = "https://files.pythonhosted.org/packages/ce/f9/a8c72619ec212eb2612730fa2b3068e2d4b59e0a0957c2e8418aa4cff59e/pillow_heif-1.3.0-cp314-cp314-win_amd64.whl", hash = "sha256:bdd6695d5be0d98ae0e9a5f88fe26f1a6eca0a5b6d43d0a92a97f89fea5842f7", size = 5640949, upload-time = "2026-02-27T12:21:16.647Z" }, + { url = "https://files.pythonhosted.org/packages/b7/31/92ce30e1ada892e18a03042bd5a8414f655304a78a36790e657f14265fed/pillow_heif-1.3.0-cp314-cp314t-macosx_10_15_x86_64.whl", hash = "sha256:65c5d05cb7f5e1eadbe9c605ae3a4dd3ef953adb33e7d809d5fb56f8a6753588", size = 4668365, upload-time = "2026-02-27T12:21:18.004Z" }, + { url = "https://files.pythonhosted.org/packages/e1/2b/789fa3c82063a780e84de667771b8ec30bc328511855f15a83a3c77011ec/pillow_heif-1.3.0-cp314-cp314t-macosx_11_0_arm64.whl", hash = "sha256:dc177fbdf598770cad4afa99c082a30b9d090e60c39656904338717803ae59b2", size = 3393554, upload-time = "2026-02-27T12:21:19.642Z" }, + { url = "https://files.pythonhosted.org/packages/5b/a4/4f8075f03c1d06d7afd674e263a3f57b7b24130c39b1544555b3b03ed369/pillow_heif-1.3.0-cp314-cp314t-manylinux_2_26_aarch64.manylinux_2_28_aarch64.whl", hash = "sha256:71f88d180547bb5112b56310c8c5e338d8358320a402c80afabc6b2f39eadddb", size = 5849609, upload-time = "2026-02-27T12:21:20.953Z" }, + { url = "https://files.pythonhosted.org/packages/0d/08/e33a10bc84ade1b4ec56bdc765735bbfd452513e33537df68107edc0eb86/pillow_heif-1.3.0-cp314-cp314t-manylinux_2_27_x86_64.manylinux_2_28_x86_64.whl", hash = "sha256:9acee893186bdde6140d30a7dc6d7c928e4ad3007989764f6e54a7a517faa332", size = 5582931, upload-time = "2026-02-27T12:21:22.571Z" }, + { url = "https://files.pythonhosted.org/packages/cb/45/6afc0f29701e0c9b911b33a35760ae6e2c581fc49b431dcce22ed18abfba/pillow_heif-1.3.0-cp314-cp314t-musllinux_1_2_aarch64.whl", hash = "sha256:7cf893689132bec18f0c55a505da9ebf3a8feb33dd354fe2ac050f20f4f862e0", size = 6891268, upload-time = "2026-02-27T12:21:24.021Z" }, + { url = "https://files.pythonhosted.org/packages/f2/0a/0d6a69f76f277692555d0e687dbf3e31d03cf76fffa3ced1fea51a18c481/pillow_heif-1.3.0-cp314-cp314t-musllinux_1_2_x86_64.whl", hash = "sha256:54404c9b6f0323114527579f54cc966b47206f99d943e47d73e1091ab0b9d2ba", size = 6515405, upload-time = "2026-02-27T12:21:25.336Z" }, + { url = "https://files.pythonhosted.org/packages/39/21/716856a36c1cc30a8f1354bf6423f251b1f50851af3e13b9cf084a13d2e3/pillow_heif-1.3.0-cp314-cp314t-win_amd64.whl", hash = "sha256:18c7c35a9d98ed9eaaf2db601ee43425ebccc698801df9c008aa04e00756a22e", size = 5641581, upload-time = "2026-02-27T12:21:26.642Z" }, ] [[package]] @@ -1763,11 +1770,11 @@ wheels = [ [[package]] name = "pytz" -version = "2025.2" +version = "2026.2" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f8/bf/abbd3cdfb8fbc7fb3d4d38d320f2441b1e7cbe29be4f23797b4a2b5d8aac/pytz-2025.2.tar.gz", hash = "sha256:360b9e3dbb49a209c21ad61809c7fb453643e048b38924c765813546746e81c3", size = 320884, upload-time = "2025-03-25T02:25:00.538Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ff/46/dd499ec9038423421951e4fad73051febaa13d2df82b4064f87af8b8c0c3/pytz-2026.2.tar.gz", hash = "sha256:0e60b47b29f21574376f218fe21abc009894a2321ea16c6754f3cad6eb7cdd6a", size = 320861, upload-time = "2026-05-04T01:35:29.667Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/81/c4/34e93fe5f5429d7570ec1fa436f1986fb1f00c3e0f43a589fe2bbcd22c3f/pytz-2025.2-py2.py3-none-any.whl", hash = "sha256:5ddf76296dd8c44c26eb8f4b6f35488f3ccbf6fbbd7adee0b7262d43f0ec2f00", size = 509225, upload-time = "2025-03-25T02:24:58.468Z" }, + { url = "https://files.pythonhosted.org/packages/ec/dd/96da98f892250475bdf2328112d7468abdd4acc7b902b6af23f4ed958ea0/pytz-2026.2-py2.py3-none-any.whl", hash = "sha256:04156e608bee23d3792fd45c94ae47fae1036688e75032eea2e3bf0323d1f126", size = 510141, upload-time = "2026-05-04T01:35:27.408Z" }, ] [[package]] @@ -2101,11 +2108,11 @@ wheels = [ [[package]] name = "types-pytz" -version = "2026.1.1.20260408" +version = "2026.2.0.20260506" source = { registry = "https://pypi.org/simple" } -sdist = { url = "https://files.pythonhosted.org/packages/f7/b7/33f5a4f29b1f285b99ff79a607751a7996194cbb98705e331dab7a2daa28/types_pytz-2026.1.1.20260408.tar.gz", hash = "sha256:89b6a34b9198ea2a4b98a9d15cbca987053f52a105fd44f7ce3789cae4349408", size = 10788, upload-time = "2026-04-08T04:28:14.54Z" } +sdist = { url = "https://files.pythonhosted.org/packages/ba/2e/a545aae2c4d2af7e3b7b967049c2f271b2f5338d96a58224fe1ee53a54f3/types_pytz-2026.2.0.20260506.tar.gz", hash = "sha256:fc6a0de6a1b7da82a748fb4065e152372dac3016559cb1eef5e8af1e338eb627", size = 10844, upload-time = "2026-05-06T05:17:51.964Z" } wheels = [ - { url = "https://files.pythonhosted.org/packages/ae/90/12c059e6bb330a22d9cc97daf027ac7fb7f50fbf518e4d88185b4d39120e/types_pytz-2026.1.1.20260408-py3-none-any.whl", hash = "sha256:c7e4dec76221fb7d0c97b91ad8561d689bebe39b6bcb7b728387e7ffd8cde788", size = 10124, upload-time = "2026-04-08T04:28:13.353Z" }, + { url = "https://files.pythonhosted.org/packages/4f/cd/df3e4ccccb2a5a0b7e59c9fb2baafb6dac0817e80799e4c9854fe4d2eba3/types_pytz-2026.2.0.20260506-py3-none-any.whl", hash = "sha256:58ab5307c20885f9bcd42ff106616eb0e32710791f8cbdc770aee2ea0c4f01fb", size = 10120, upload-time = "2026-05-06T05:17:51.026Z" }, ] [[package]] diff --git a/website/data/faq.yaml b/website/data/faq.yaml index 797e6966b..3948afe31 100644 --- a/website/data/faq.yaml +++ b/website/data/faq.yaml @@ -73,7 +73,7 @@ items: - question: How do I rotate the screen for portrait orientation? answer: | - Rotation happens at the kernel / firmware level. The Anthias viewer renders straight to the Linux framebuffer (Qt `linuxfb`) and to KMS (mpv `--vo=drm`) — there's no Wayland compositor in the stack — so the standard Raspberry Pi config knobs apply directly. + Use the Settings page in the Anthias dashboard ("Screen rotation"). On Pi 5 / x86 / arm64 the viewer runs under the `cage` kiosk Wayland compositor and the rotation is applied via `wlr-randr`. On Pi 2 / Pi 3 / Pi 4 the viewer renders to the Linux framebuffer directly and the transform is applied inside the media player; the Settings page works the same way. Edit `/boot/firmware/config.txt` and add one of: @@ -89,7 +89,7 @@ answer: | Anthias displays at whatever resolution the OS reports — there's no resolution setting in the dashboard. Set the output mode at the OS level: on Raspberry Pi OS, use `sudo raspi-config` → **Display Options** → **Resolution**, or set `hdmi_group` / `hdmi_mode` in `/boot/firmware/config.txt`. - 4K is supported on hardware that supports it natively: Pi 5 handles 4K@60Hz on either HDMI port; Pi 4 supports 4K@30Hz on `HDMI-0` (or 4K@60Hz with `hdmi_enable_4kp60=1` and adequate cooling). + 4K is supported on Pi 5 / x86 / arm64 — those run the viewer under the `cage` Wayland compositor and the GPU handles the upscale for free. On Pi 4 the viewer renders to the Linux framebuffer directly and mpv hands video scaling to the V3D, so 1080p source on a 4K connector works smoothly; demanding 4K-source video on Pi 4 will drop frames, however, because the V3D doesn't have the bandwidth headroom Pi 5's V3D 7.1 has. - question: My screen is black or stuck on "Manage the content" — what now? answer: | @@ -98,7 +98,7 @@ 1. SSH in and run `docker compose ps` from `~/anthias/` — all four containers (`anthias-server`, `anthias-celery`, `anthias-viewer`, `redis`) should be `Up`. 2. If a container is restarting, tail its logs: `docker logs -f anthias-anthias-viewer-1` (or `…-server-1`). 3. Open the dashboard from another device and confirm at least one asset is **enabled** with a current schedule. - 4. If the display has gone to power save, briefly unplug and reconnect the HDMI cable, or check the TV's input source. Anthias renders to the framebuffer (Qt `linuxfb`) and uses no X server, so `xset` won't help. + 4. If the display has gone to power save, briefly unplug and reconnect the HDMI cable, or check the TV's input source. Anthias renders directly to KMS (via `cage` on Pi 5 / x86 / arm64, or Qt `linuxfb` on Pi 2 / Pi 3 / Pi 4) and uses no X server, so `xset` won't help. If a recent update broke the display, running `~/anthias/bin/upgrade_containers.sh` re-creates the containers cleanly.