remote code execution via SVC layer context handling with attacker-controlled frames

Summary

A remote code execution vulnerability was found in libaom, the reference AV1 codec implementation. Insufficient bounds validation in the AV1 encoder's SVC (Scalable Video Coding) layer ID control allows an attacker to supply crafted video frame pixels that overlap with internal encoder layer context structures. In fork-based video processing services, an attacker can use this to hijack the cyclic refresh map pointer, brute-force the process base address via a crash oracle, and redirect control flow to achieve arbitrary command execution. Exploitation requires the target service to use libaom with SVC encoding enabled and accept attacker-supplied video frames. A remote code execution vulnerability was found in libaom, the reference AV1 codec implementation. Insufficient bounds validation in the AV1 encoder's SVC (Scalable Video Coding) layer ID control allows an attacker to supply crafted video frame pixels that overlap with internal encoder layer context structures. In fork-based video processing services, an attacker can use this to hijack the cyclic refresh map pointer, brute-force the process base address via a crash oracle, and redirect control flow to achieve arbitrary command execution. Exploitation requires the target service to use libaom with SVC encoding enabled and accept attacker-supplied video frames. This vulnerability is rated as Critical severity because the researcher demonstrated successful remote code execution against a fork-based video processing service. The exploit chain leverages attacker-controlled pixel values to hijack internal encoder pointers, uses a crash oracle to brute-force ASLR, and ultimately achieves arbitrary command execution. However, the attack complexity is elevated: it requires a fork-based service architecture (for the crash oracle), multiple encoding attempts (for ASLR brute-force), and knowledge of the target binary layout. In Red Hat products, libaom ships bundled within Firefox and Thunderbird. Firefox does not use a fork-based architecture for WebRTC encoding, and SVC layer parameters are managed internally, making the demonstrated exploit chain not directly applicable to the browser context. RHEL-AI 3.4 (aom 3.12.0) and Hummingbird 1 (aom 3.13.3) ship standalone libaom packages within the affected version range. Fork-based transcoding or video conferencing services that use libaom with SVC encoding and accept attacker-supplied frames are at highest risk for this specific exploit chain. Red Hat severity: Important — CVSS 7.1 (CVSS:3.1/AV:N/AC:H/PR:N/UI:R/S:U/C:L/I:H/A:H). Weakness: CWE-787. Fixed by RHSA-2026:30814 — update the affected packages (`sudo dnf update`). Affected Red Hat products: Red Hat Hardened Images.