ANT-2026-CAH7XGSQ · lua/lua

stack-overflow low

Severity Claude low · Security research firm low · Maintainer -

Discovered by Claude Mythos Preview

REPORT

Anthropic's analysis, sealed at approval. Disclosure to the maintainer was performed by Anvil Security.

ANT-2026-CAH7XGSQ: stack-overflow (C stack exhaustion via unbounded recursion) — LOW VALUE tier.

The match() function in lstrlib.c recurses during pattern matching and is protected by a matchdepth counter initialized to MAXCCALLS=200. When a pattern error is caught by pcall, the counter is not properly restored, so subsequent pattern operations recurse without bound. ASAN confirms ~250 recursive frames before the guard page triggers SIGSEGV, and UBSAN separately confirms the counter wraps past INT_MIN. An attacker who can run Lua code or supply pattern strings achieves deterministic denial of service; the guard page prevents escalation to memory corruption.

Target

Project: lua
Location: lstrlib.c:573

Technical Details

ASAN: "ERROR: AddressSanitizer: stack-overflow on address 0x7ffc43d9aff8" — the trace shows ~250 self-recursive frames of match(), far past the MAXCCALLS=200 limit that should have thrown an error. The root cause is that the matchdepth field in MatchState is not reset when a prior pattern error unwinds through pcall, so the decrement-and-check at function entry operates on stale state and never fires. The secondary UBSAN hit ("signed integer overflow: -2147483648 - 1") corroborates that the counter has been decrementing unchecked across many invocations.

Crash trace (truncated — full trace in attached crash.log):

=== Production binary (/usr/local/bin/lua) ===
Segmentation fault
EXIT=139

=== ASAN build (/workspace/repo/lua) ===
AddressSanitizer:DEADLYSIGNAL
=================================================================
==347==ERROR: AddressSanitizer: stack-overflow on address 0x7ffc43d9aff8 (pc 0x55d511094374 bp 0x7ffc43d9b030 sp 0x7ffc43d9afe0 T0)
    #0 0x55d511094374 in match /workspace/repo/lstrlib.c:573
    #1 0x55d5110951bb in match /workspace/repo/lstrlib.c:645
    #2 0x55d5110951bb in match /workspace/repo/lstrlib.c:645
    ... [~250 frames of match /workspace/repo/lstrlib.c:645] ...
SUMMARY: AddressSanitizer: stack-overflow /workspace/repo/lstrlib.c:573 in match

Reproduction

Invoke a pattern-matching call that raises an error inside match(), caught by pcall, leaving matchdepth unrestored
Repeat to drive the residual matchdepth value negative (optionally to INT_MIN for the UBSAN variant)
Invoke a pattern with a recursive construct; the depth check never triggers because the counter is already below zero
match() self-recurses ~250+ frames until the C stack hits the guard page and the process receives SIGSEGV

[No reproducer or sanitizer output attached — request from security-cvd@anthropic.com if needed.]

Acknowledgement

This vulnerability was discovered by Claude, Anthropic's AI assistant, and triaged by the Anthropic security team in collaboration with Anthropic Research. Please direct questions to security-cvd@anthropic.com and reference ANT-2026-CAH7XGSQ.

Reference: ANT-2026-CAH7XGSQ
Anthropic CVD Policy: https://www.anthropic.com/coordinated-vulnerability-disclosure

SECURITY RESEARCH FIRM ANALYSIS

Triage and disclosure were performed by Anvil Security.

Verdict: true positive
Severity: low

UPSTREAM FIX

The change that resolved this finding.

diff --git a/lstrlib.c b/lstrlib.c
index 874cec8086..dd3c0fd08c 100644
--- a/lstrlib.c
+++ b/lstrlib.c
@@ -757,19 +757,25 @@ static int nospecials (const char *p, size_t l) {
 }
 
 
+/*
+** Prepare state for matches. These fields are not affected by each match.
+*/
 static void prepstate (MatchState *ms, lua_State *L,
                        const char *s, size_t ls, const char *p, size_t lp) {
   ms->L = L;
-  ms->matchdepth = MAXCCALLS;
   ms->src_init = s;
   ms->src_end = s + ls;
   ms->p_end = p + lp;
 }
 
 
+/*
+** (Re)prepare state for a match, setting fields that change during
+** each match.
+*/
 static void reprepstate (MatchState *ms) {
+  ms->matchdepth = MAXCCALLS;
   ms->level = 0;
-  lua_assert(ms->matchdepth == MAXCCALLS);
 }
 
 
diff --git a/testes/pm.lua b/testes/pm.lua
index 720d2a3562..feab33dbbe 100644
--- a/testes/pm.lua
+++ b/testes/pm.lua
@@ -347,6 +347,16 @@ do   -- init parameter in gmatch
 end
 
 
+do  -- bug since 5.3
+  local N = 20000
+  local iter = string.gmatch(string.rep("a", N), string.rep("a?", N))
+  pcall(iter)   -- error for pattern too complex
+  -- calling function again found recursion count ('matchdepth') equal
+  -- to -1, so it did not detect next C-stack overflow
+  pcall(iter)
+end
+
+
 -- tests for `%f' (`frontiers')
 
 assert(string.gsub("aaa aa a aaa a", "%f[%w]a", "x") == "xaa xa x xaa x")

https://github.com/lua/lua/commit/efddc2309c5ff8a1842bea8a9c0d7d4a5d6e1e60

TIMELINE

Dates from discovery through public reveal.

2026-03-20 Reported to tracker
2026-03-31 Patch released
2026-04-09 Sent to maintainer
2026-05-19 Maintainer acknowledged
2026-08-17 Publicly revealed

PROVENANCE

SHA-3-512 hash:

b29364e75f34a3b4b7e66d5461c338ca5e875426bdf8210361f5b47562d73733b1a51aa88d15858d2065e5809f5f5e296a2d5248703efd326e53afa0ed32ab8e

Committed 2026-04-09 11:50 PT

Revealed 2026-08-17 10:47 PT

Verify (download preimage.json)

Show preimage JSON

{
  "ant_id": "ANT-2026-CAH7XGSQ",
  "bug_class": "stack-overflow",
  "claude_severity": "low",
  "commit_sha": null,
  "created_at": "2026-03-20T19:13:27+00:00",
  "description": "The match() function in lstrlib.c recurses during pattern matching and is protected by a matchdepth counter initialized to MAXCCALLS=200. When a pattern error is caught by pcall, the counter is not properly restored, so subsequent pattern operations recurse without bound. ASAN confirms ~250 recursive frames before the guard page triggers SIGSEGV, and UBSAN separately confirms the counter wraps past INT_MIN. An attacker who can run Lua code or supply pattern strings achieves deterministic denial of service; the guard page prevents escalation to memory corruption.",
  "discovered_at": null,
  "location": "lstrlib.c:573",
  "poc_sha256": "867a98c87e1c062c035d2ed9525b6b66bcd36e93d1559869c9da25e22629bf56",
  "preimage_version": 1,
  "project": "lua",
  "reproduction": [
    "1. Invoke a pattern-matching call that raises an error inside match(), caught by pcall, leaving matchdepth unrestored",
    "2. Repeat to drive the residual matchdepth value negative (optionally to INT_MIN for the UBSAN variant)",
    "3. Invoke a pattern with a recursive construct; the depth check never triggers because the counter is already below zero",
    "4. match() self-recurses ~250+ frames until the C stack hits the guard page and the process receives SIGSEGV"
  ],
  "technical_details": "ASAN: \"ERROR: AddressSanitizer: stack-overflow on address 0x7ffc43d9aff8\" — the trace shows ~250 self-recursive frames of match(), far past the MAXCCALLS=200 limit that should have thrown an error. The root cause is that the matchdepth field in MatchState is not reset when a prior pattern error unwinds through pcall, so the decrement-and-check at function entry operates on stale state and never fires. The secondary UBSAN hit (\"signed integer overflow: -2147483648 - 1\") corroborates that the counter has been decrementing unchecked across many invocations.",
  "title": "stack-overflow (C stack exhaustion via unbounded recursion) — LOW VALUE tier.",
  "vendor_severity": "low"
}