Intermittent segmentation fault in System.Security.Cryptography.AesGcm on Fedora 41 with .NET 9 (crash in ossl_aes_gcm_decrypt_avx512)

[mbtolou]

text### Description

A framework-dependent .NET 9 application using System.Security.Cryptography.AesGcm (or any crypto that triggers AES-GCM decryption) experiences random segmentation faults every few hours (typically 2-12 hours) on Fedora Linux 41.

The crash occurs in OpenSSL's AVX-512 path for AES-GCM decryption:
Process ... (dotnet) crashed in ossl_aes_gcm_decrypt_avx512()
textStack trace from core dump (via abrt-notification):
#21 0x00005606b703fefe _Z9exe_startiPPKc (dotnet + 0x7efe)
#22 0x00005606b704022f main (dotnet + 0x822f)
#25 0x00005606b703ebe5 _start (dotnet + 0x6be5)
textJournalctl logs:
audit[PID]: ANOM_ABEND ... exe="/usr/lib64/dotnet/dotnet" sig=11
systemd-coredump: Process ... (dotnet) ... terminated abnormally with signal 11/SEGV
textThis is intermittent and only happens under load when AES-GCM operations are performed (likely decryption).

Environment

• OS: Fedora Linux 41 (Server Edition)
• Kernel: 6.17.10-100.fc41.x86_64
• .NET: 9.0.11 (runtime) / 9.0.112 (SDK) – installed from Fedora repositories
• OpenSSL: OpenSSL 3.2.6 30 Sep 2025
• CPU: Supports full AVX-512 (including avx512f, avx512dq, avx512ifma, vaes, vpclmulqdq, avx512_vnni, etc.)
• App publish: Framework-dependent (dotnet MyApp.dll), built with dotnet build -c release
• No memory leak; app runs stably otherwise.

Workaround

Setting the following environment variable prevents the crash by disabling VAES/VPCLMULQDQ (fastest AVX-512 paths for GCM):
export OPENSSL_ia32cap="~0x20000000000:~0x40000000000"
textDisabling full AVX-512 also works:
export OPENSSL_ia32cap="~0x2000000000000000"
textThis forces fallback to slower but stable paths (AVX2/AES-NI).

Publishing as self-contained may also avoid it (untested).

Reproduction

Hard to reproduce deterministically due to intermittency, but runs for hours with AES-GCM usage (e.g., encrypted network traffic or data processing) and eventually crashes.

No similar issues found in existing dotnet/runtime issues.

Possible cause

Likely a bug or instability in OpenSSL 3.2.6's AVX-512 AES-GCM implementation (specifically decrypt path) on certain CPUs. .NET delegates AES-GCM to system OpenSSL on Linux.

Suggest considering a runtime workaround (e.g., detect AVX-512 and disable problematic paths via OPENSSL_ia32cap, or fallback to managed implementation).

[dotnet-policy-service]

Tagging subscribers to this area: @dotnet/area-system-security, @bartonjs, @vcsjones
See info in area-owners.md if you want to be subscribed.

[vcsjones]

cc @tmds Is this an issue y'all would happen to know about?

fallback to managed implementation

.NET does not have a managed implementation (and can't, by policy). So the only viable work around here is to disable the capabilities in OpenSSL. Whether .NET should try to work around it is another issue.

I suppose a place to start is trying to understand if this is a bug in OpenSSL and if it's been fixed.

[jkotas]

core dump

Could you please share disassembly of the method that it crashed in that shows where it crashed exactly?

[mbtolou]

If there are specific instructions for doing this, please let me know—I’ll follow them and send you the results.

That said, I’ve already resolved the issue using Grok’s guidance: I disabled OpenSSL’s AVX-512-related instructions, and the problem no longer occurs.

Reproducing the issue again in the production environment would be problematic for me.

Nevertheless, I’ll do my best to carry this out at an appropriate opportunity.

[jkotas]

If there are specific instructions for doing this, please let me know

1. Open the core dump in gdb or lldb debugger (gdb -q /path/to/executable /path/to/core)
2. Disassemble around the crash location (x/80i $pc-40 - this is the command for gdb)

(You can use your favorite llvm to get more detailed or customized instructions as needed.)