No Spam Proxy GCM Bug

/ Martijn Brinkers

When S/MIME GCM Goes Wrong: “mac check in GCM failed” with No Spam Proxy

We recently received two bug reports where a CipherMail Gateway was unable to decrypt an incoming S/MIME message. In both cases the failure was caused by an integrity check error during decryption:

javax.crypto.AEADBadTagException: mac check in GCM failed

Both messages originated from No Spam Proxy.

The odd part: “Same email,” different outcome

The affected customer had just upgraded from CipherMail v5 to v6. Before the upgrade, they had exchanged encrypted mail with the same external sender without any issues.

What made this case especially confusing:

  1. The customer received two separate emails with the same Message-ID
  2. One was delivered to a To: recipient, the other to a CC: recipient
  3. One message decrypted successfully, while the other, apparently “the same” message, failed with the GCM MAC/checksum error

Why the error points to GCM

“mac check in GCM failed” typically means the authentication tag validation failed during AES-GCM decryption. In other words: the ciphertext’s integrity protection did not validate, so the decryptor correctly refuses to output plaintext. Galois/Counter Mode (GCM) is widely used and well established, but its use in S/MIME is comparatively newer than classic AES-CBC-based S/MIME encryption.

CipherMail v5 vs v6: the key difference

  1. CipherMail ≤ v5 did not support S/MIME with AES-GCM by default (there was a patch, but it wasn’t part of the standard product build).
  2. CipherMail v6 introduced full support for S/MIME GCM.

That matters because S/MIME messages can include (and clients/gateways often learn) capabilities, i.e., which algorithms a party supports. If a system learns that a recipient supports AES-GCM, it may prefer it over older modes like AES-CBC.

What likely happened: algorithm selection per recipient

Based on the behavior, the most plausible explanation is:

  1. While running CipherMail v5, the gateway did not advertise GCM capability.
  2. No Spam Proxy therefore continued to use AES-CBC, and everything worked.
  3. After upgrading to CipherMail v6, No Spam Proxy learned (via signed mail and capability information) that some recipients now support AES-GCM.
  4. No Spam Proxy then started using AES-GCM for certain recipients, but still used AES-CBC for others (depending on which capabilities it had already observed per user).

Because a single S/MIME encrypted payload can’t be both CBC and GCM at the same time, No Spam Proxy likely split the delivery into two messages (hence the same Message-ID appearing twice), encrypting each copy using the algorithm it believed matched the recipient’s capabilities.

Reproducing the issue: it depends on No Spam Proxy version

Our first assumption in interoperability issues is always: we might be doing something wrong. That’s also the easiest kind of problem to fix.

To diagnose this properly, we needed to reproduce it with messages encrypted to a keypair we control. That required setting up No Spam Proxy and running controlled tests with GCM.

  1. With a modern No Spam Proxy setup, GCM decryption worked.
  2. But the customer reports were consistent, and happened twice, so we suspected a version-specific issue.

We then tested the last No Spam Proxy 14.x release (14.2) and were able to reproduce the failure. That strongly suggested the problem had been fixed somewhere between 14.2 and 15.7.

The smoking gun: “bad mac” fixed in No Spam Proxy 15.5

Once we knew it was likely a fixed bug, we found an item in the No Spam Proxy release notes (fixed in 15.5):

43883 Behebung des „bad mac“-Fehlers bei aes-gcm-Verschlüsselung.

(“Fixing the ‘bad mac’ error in aes-gcm encryption”)

Source:

https://docs.nospamproxy.com/Server/15/Suite/de-de/Content/release-notes/nospamproxy-server-15-5-0-1377.htm

The note doesn’t specify whether it affects TLS, S/MIME, or another subsystem, but the symptoms match.

Is it only CipherMail interoperability? We checked OpenSSL

To rule out a “CipherMail-only” edge case, we also tested whether OpenSSL could decrypt the same messages:

  1. Email produced by No Spam Proxy 15.7: decrypts successfully with OpenSSL
  2. Email produced by No Spam Proxy 14.2: fails to decrypt with OpenSSL

$ openssl cms -decrypt -inkey testCertificates.pem -in nsp-14_2-aes-256-gcm-encrypt-only.eml

Error decrypting CMS structure 40C75158277F0000:error:06800079:asn1 encoding routines:asn1_item_embed_d2i:field missing:../crypto/asn1/tasn_dec.c:465:Field=num, Type=asn1_oct_int 40C75158277F0000:error:0680006E:asn1 encoding routines:ASN1_item_unpack:decode error:../crypto/asn1/asn_pack.c:59: 40C75158277F0000:error:0680006D:asn1 encoding routines:ossl_asn1_type_get_octetstring_int:data is wrong:../crypto/asn1/evp_asn1.c:183: 40C75158277F0000:error:0300007A:digital envelope routines:evp_cipher_asn1_to_param_ex:cipher parameter error:../crypto/evp/evp_lib.c:215: 40C75158277F0000:error:17000066:CMS routines:ossl_cms_EncryptedContent_init_bio:cipher parameter initialisation error:../crypto/cms/cms_enc.c:103:

That strongly indicates the older No Spam Proxy versions generated AES-GCM S/MIME that was not generally interoperable, even with the de-facto baseline toolchain.

What to do if you hit this

If you see decryption failures and your logs contain:

mac check in GCM failed

…and the sender uses No Spam Proxy, the fix is on the sender side. There are two practical options:

  1. Upgrade No Spam Proxy (recommended; the issue appears resolved in 15.5+), or
  2. If they cannot upgrade, configure a partner/domain setting in No Spam Proxy and explicitly force AES-128 (compatible) (i.e., AES-CBC) for that partner domain.

This avoids AES-GCM entirely for that delivery path and restores compatibility with gateways that would otherwise receive broken GCM-encrypted S/MIME.

No Spam Proxy Partner Settings