heap buffer overflow in AV1 encoder first-pass stats buffer via LAP mode

Summary

A heap buffer overflow vulnerability was found in libaom, the reference AV1 codec implementation. A flaw in the AV1 encoder's Look-Ahead Processing (LAP) mode causes the first-pass stats ring buffer wrap-around guard to be bypassed when g_lag_in_frames is set to 1 or higher. This results in a 232-byte out-of-bounds write on every encoded frame after the second, corrupting adjacent heap objects. An attacker who can influence encoder configuration in a transcoding service or WebRTC session could exploit this to cause a denial of service (process crash) or potentially achieve code execution. A heap buffer overflow vulnerability was found in libaom, the reference AV1 codec implementation. A flaw in the AV1 encoder's Look-Ahead Processing (LAP) mode causes the first-pass stats ring buffer wrap-around guard to be bypassed when g_lag_in_frames is set to 1 or higher. This results in a 232-byte out-of-bounds write on every encoded frame after the second, corrupting adjacent heap objects. An attacker who can influence encoder configuration in a transcoding service or WebRTC session could exploit this to cause a denial of service (process crash) or potentially achieve code execution. This vulnerability is rated as Important severity because a heap buffer overflow with attacker-influenced data can cause reliable denial of service and potentially lead to code execution, though the attacker has only indirect control over the written values (encoder-computed statistics). In Red Hat products, libaom ships bundled within Firefox and Thunderbird as a statically-linked dependency used for AV1 decoding and WebRTC encoding. The vulnerable code path requires the encoder to be configured with g_lag_in_frames >= 1 (Look-Ahead Processing mode). In Firefox's WebRTC implementation, the encoder configuration is controlled by the browser itself and not exposed to remote peers, which significantly limits the attack surface compared to standalone transcoding services. RHEL-AI 3.4 and Hummingbird 1 ship standalone libaom (aom) packages at versions within the affected range. Applications on those platforms that use the libaom encoder API with LAP mode and accept untrusted configuration input are vulnerable. Red Hat severity: Important — CVSS 7.6 (CVSS:3.1/AV:N/AC:L/PR:N/UI:R/S:U/C:L/I:L/A:H). Weakness: CWE-122. Fixed by RHSA-2026:30814 — update the affected packages (`sudo dnf update`). Affected Red Hat products: Red Hat Hardened Images.