CheckTLS shows hostname verify error but OpenSSL does not?

QUESTION

Before configuring enforced SMTP/TLS for specific recipient domains I usually check the configuration of that recipient domain's mail server with checktls.com.

Now I have a case which I do not understand: xxxOur.com [ed note: all domain names changed for privacy]

MX records are pointing to servers with domain name ending with .xxxMX.com The certificates CN = *.xxxCN.com

checktls.com states: [001.084] Cert VALIDATED: [001.084] Cert Hostname DOES NOT VERIFY (mx5.xxxMX.com != *.xxxCN.com | DNS:*.xxxCN.com | DNS:xxxCN.com) [001.084] (see RFC-2818 section 3.1 paragraph 4 for info on wildcard ("*") matching) [001.084] So email is encrypted but the host is not verified

But when I test with openssl I receive: Verify return code: 0 (ok)

And when I enforce SMTP/TLS (it's a sendmail MTA) for domain xxxOurDomain.com, mails are being delivered and the log says: tls_verify=OK

I can see the following SMTP banner and response from the SMTP server in the greeting section after the EHLO submitted by the SMTP client: 250-mx1.xxxCN.com

Does this mean that it's sufficient if a certificate CN matches the SMTP banner for openssl to regard an SMTP/TLS connection as verified?

This looks somewhat confusing to me.

Can you provide me some more information about this specific topic? I'm especially interested in what names are involved in certificate verification during an SMTP/TLS handshake.

ANSWER

You are asking how various programs use three different hostnames:

MX Host

The host (mx1.xxxMX.com) identified to receive email for a domain (xxxOur.com). It comes from looking up the recipient's domain in the Internet Domain Name System.

CN Entry

The host or host-wildcard (xxxCN.com) given in the SSL Certificate of the MX Host.

HELO/EHLO Host

The hostname returned in response to an SMTP HELO/EHLO command

HELO/EHLO Host

A site can return any hostname in their HELO/EHLO response, so it is not a reliable security constraint. However many mailers will log a security warning when they find a mis-match between what they are expecting in the HELO/EHLO response and what they receive.

The more detailed Output Format levels of the CheckTLS Test To: (TestReceiver) test show the HELO/EHLO response but CheckTLS does not try to match it against anything and it does not affect the Confidence Factor.

CN Entry

The CN Entry is a field included in a site's SSL Certificate. Because the certificate is encrypted and should be signed by a trusted Certificate Authority, the CN Entry is a reliable security constraint.

The CheckTLS Test To: (TestReceiver) test does use the CN Entry as part of its hostname verification and a mis-match does affect the Confidence Factor.

MX Host

The MX Host comes from a DNS lookup. A DNS lookup is not completely secure, and there are projects working on making it more secure, but MX Hosts from DNS lookups are considered a reliable security constraint.

The CheckTLS Test To: (TestReceiver) test does use the MX Host as part of its hostname verification and a mis-match does affect the Confidence Factor.

But Why Does CheckTLS Complain When OpenSSL and My Mailer Do Not?

Current versions of openssl do not check for any hostname matches. It reports the name but does not use it. You can see this for yourself using:

openssl s_client -connect mail11-do.checktls.com:25 -starttls smtp -servername baddomain.com

This tells openssl to connect to one of our MX Hosts and to expect it to be named "baddomain.com". openssl happily and without reporting anything out of the ordinary connects to our host, even though it clearly the host it connected to (mail11-do.checktls.com) has nothing to do with baddomain.com.

Most mailers do check that the MX Host match the CN Entry, but they just log a warning (which no one ever sees) and continue to send the email. This is consistent with the traditional Internet mantra of "the data must get through".

Some mailers can be configured, or may even be configured out of the box, to fail if the MX Host does not match the CN Entry. Security is becoming more prevalent in today's world.

Thank you CheckTLS!

CheckTLS does compare the two reliable security constraint names.

Since a mis-match typically does not prevent email from going through, CheckTLS only warns about mis-matches (in yellow) and includes a note that email will be encrypted (i.e. safe from prying eyes) but there is a small chance it is being sent to the wrong place. And since it is their certificate that is encrypting the email, they will be able to read it.

CheckTLS also helps diagnose if your mailer is configured to fail mail when the MX does not match the CN. This can happen without a mail administrator's knowledge, say with a software upgrade, or a new trading partner. Suddenly email stops working, and it is not clear why. CheckTLS shows you the problem.