ANT-2026-RNHRV8B9 · libreoffice/core

stack-buffer-overflow medium

CVE-2026-63275

Severity Claude high · Security research firm high · Maintainer medium

Discovered by Claude Mythos Preview

REPORT

Anthropic's analysis, sealed at approval. Disclosure to the maintainer was performed by Ada Logics.

ANT-2026-RNHRV8B9: CFF hint-stack bound compared against double the array size

In LibreOffice's CFF font subsetter (vcl/source/fontsubset/cff.cxx), addHints() guards the hint stack with (mnHintSize + mnStackIdx) > 2*NMAXHINTS but mnHintStack has only NMAXHINTS (192) slots, so the guard permits twice the actual capacity. A glyph charstring that issues repeated hstemhm/vstemhm operators accumulates mnHintSize past 192 and writes up to ~192 attacker-controlled doubles (~1.5KB) past the array. CffContext is a stack automatic in ConvertCFFfontToType1(), so the overflow clobbers maCharWidth, mbDoSeac, the std::vector maExtraGlyphIds (whose corrupted internal pointers give a write/free primitive), and the calling stack frame. The path is reached during routine PDF export of any document embedding a CFF/OTF font (pdfwriter_impl.cxx → PhysicalFontFace::CreateFontSubset → ConvertCFFfontToType1), including headless soffice --convert-to pdf pipelines, and hb_subset preserves the hint operators.

Target

Project: libreoffice/core
Location: vcl/source/fontsubset/cff.cxx:867
Discovery: static analysis — not yet dynamically reproduced

Technical Details

NMAXHINTS is defined as 296 = 192 and mnHintStack is declared with NMAXHINTS elements, but the guard at line 867 checks against 2NMAXHINTS (384) — the author double-counted the hint-pair factor already baked into the constant. Because mnHintSize persists across every hstem/vstem/hintmask operator within a glyph, five hstemhm ops with 48 operands each drive mnHintSize to 240, under the erroneous 384 guard but past the 192-slot array, and the loop mnHintStack[mnHintSize++] = nHintOfs writes attacker-chosen ValType doubles off the end of the stack object.

Reproduction

  1. Craft an OTF/CFF font with a glyph charstring containing repeated hstemhm/vstemhm operators accumulating >192 (up to 384) hint values
  2. Embed the font in a document (ODT/DOCX/etc.) and deliver it to the victim or upload to a headless conversion service
  3. Trigger PDF export; pdfwriter_impl.cxx calls PhysicalFontFace::CreateFontSubset → ConvertCFFfontToType1 on the embedded font
  4. convert2Type1Ops processes the glyph, addHints() passes the faulty 2*NMAXHINTS guard and writes attacker doubles past mnHintStack[192]
  5. Overwritten std::vector internal pointers are dereferenced on push_back/destruction, yielding a write/free primitive before any stack canary check

[No reproducer or sanitizer output attached — request from security-cvd@anthropic.com if needed.]

Suggested Fix

Change the capacity check in addHints() to compare against the actual array size (NMAXHINTS, not 2*NMAXHINTS) so the total number of stored hint values can never exceed the mnHintStack allocation.

Acknowledgement

This vulnerability was discovered by Claude, Anthropic's AI assistant, and triaged by the Anthropic security team in collaboration with Anthropic Research. Please direct questions to security-cvd@anthropic.com and reference ANT-2026-RNHRV8B9.

Reference: ANT-2026-RNHRV8B9
Anthropic CVD Policy: https://www.anthropic.com/coordinated-vulnerability-disclosure

SECURITY RESEARCH FIRM ANALYSIS

Triage and disclosure were performed by Ada Logics.

Verdict
true positive
Severity
high
UPSTREAM FIX

The change that resolved this finding.

diff --git a/vcl/source/fontsubset/cff.cxx b/vcl/source/fontsubset/cff.cxx
index 0aa33be342593..fb1409fcacacd 100644
--- a/vcl/source/fontsubset/cff.cxx
+++ b/vcl/source/fontsubset/cff.cxx
@@ -1472,7 +1472,7 @@ bool CffContext::addHints(bool bVerticalHints)
     if (mnStackIdx & 1)
         --mnStackIdx; //#######
 
-    if ((mnHintSize + mnStackIdx) > 2 * NMAXHINTS)
+    if (o3tl::make_unsigned(mnHintSize + mnStackIdx) > std::size(mnHintStack))
         return false;
 
     ValType nHintOfs = 0;

https://github.com/LibreOffice/core/commit/1db305f340787ab7e59d089e51ac9d6fc07c2715

TIMELINE

Recorded dates, in order.

  1. 2026-04-02 Discovered or logged
  2. 2026-07-04 Sent to maintainer
  3. 2026-07-24 Patch released
  4. 2026-08-12 Maintainer acknowledged
  5. 2026-09-28 Publicly revealed
PROVENANCE

SHA-3-512 hash:

2f4789bd452bfa96bea6d71f747bcaffcfe45f9cc4d83ece8f4cb87f0ce7f68e1186083314d35ad9a564b230684ba9fead02f1f1fcb2b6d2a0f62a5df090bcd7

Committed 2026-07-22 07:32 UTC

Revealed 2026-09-28 21:06 UTC

Verify (download preimage.json)

Show preimage JSON 
{
  "ant_id": "ANT-2026-RNHRV8B9",
  "bug_class": "Stack Buffer Overflow / Off-by-N",
  "claude_severity": "high",
  "commit_sha": null,
  "created_at": "2026-04-16T01:54:45+00:00",
  "description": "In LibreOffice's CFF font subsetter (vcl/source/fontsubset/cff.cxx), addHints() guards the hint stack with `(mnHintSize + mnStackIdx) > 2*NMAXHINTS` but mnHintStack has only NMAXHINTS (192) slots, so the guard permits twice the actual capacity. A glyph charstring that issues repeated hstemhm/vstemhm operators accumulates mnHintSize past 192 and writes up to ~192 attacker-controlled doubles (~1.5KB) past the array. CffContext is a stack automatic in ConvertCFFfontToType1(), so the overflow clobbers maCharWidth, mbDoSeac, the std::vector maExtraGlyphIds (whose corrupted internal pointers give a write/free primitive), and the calling stack frame. The path is reached during routine PDF export of any document embedding a CFF/OTF font (pdfwriter_impl.cxx → PhysicalFontFace::CreateFontSubset → ConvertCFFfontToType1), including headless `soffice --convert-to pdf` pipelines, and hb_subset preserves the hint operators.",
  "discovered_at": "2026-04-02T00:00:00+00:00",
  "location": "vcl/source/fontsubset/cff.cxx:867",
  "poc_sha256": null,
  "preimage_version": 1,
  "project": "LibreOffice/core",
  "reproduction": [
    "1. Craft an OTF/CFF font with a glyph charstring containing repeated hstemhm/vstemhm operators accumulating >192 (up to 384) hint values",
    "2. Embed the font in a document (ODT/DOCX/etc.) and deliver it to the victim or upload to a headless conversion service",
    "3. Trigger PDF export; pdfwriter_impl.cxx calls PhysicalFontFace::CreateFontSubset → ConvertCFFfontToType1 on the embedded font",
    "4. convert2Type1Ops processes the glyph, addHints() passes the faulty 2*NMAXHINTS guard and writes attacker doubles past mnHintStack[192]",
    "5. Overwritten std::vector<sal_GlyphId> internal pointers are dereferenced on push_back/destruction, yielding a write/free primitive before any stack canary check"
  ],
  "technical_details": "NMAXHINTS is defined as 2*96 = 192 and mnHintStack is declared with NMAXHINTS elements, but the guard at line 867 checks against 2*NMAXHINTS (384) — the author double-counted the hint-pair factor already baked into the constant. Because mnHintSize persists across every hstem/vstem/hintmask operator within a glyph, five hstemhm ops with 48 operands each drive mnHintSize to 240, under the erroneous 384 guard but past the 192-slot array, and the loop `mnHintStack[mnHintSize++] = nHintOfs` writes attacker-chosen ValType doubles off the end of the stack object.",
  "title": "CFF hint-stack bound compared against double the array size",
  "vendor_severity": "high"
}