Outlook 2007 MD5 SMTP auth broken

I’ve been playing a lot with Office 2007 lately in my bid to get familiar with it for offering technical support, and as everything communication wise for me revolves around email I have been giving Outlook 2007 a pounding - especially since Outlook has a very poor track record when it comes to large IMAP mailboxes. In doing so, there are a number of issues I have encountered, however the biggest would be that any SMTP auth that uses MD5 will fail. It seems that Outlook 2007 is only successful when using LOGIN or PLAIN auth attempts when trying to send mail.

If you disable LOGIN and PLAIN authentication steps in your SMTP server, you will no longer be able to send mail - even though MD5 auth is available. In fact, to make things even more annoying, Outlook 2007 will attempt to use MD5 auth - and fail. I have used MD5 SMTP auth for years with various mail clients (mainly Mail on OSX) without any issues, so it took me a number of hours using ethereal to find out what was going on.

When you connect to an SMTP server, most mail clients say EHLO (vs the older HELO) to introduce themselves and also get the SMTP servers capabilities. This connection usually looks a bit like this:

$ telnet localhost 25
Trying 127.0.0.1...
Connected to localhost (127.0.0.1).
Escape character is '^]'.
220 my.mail.server.com ESMTP Sendmail 8.13.8/8.13.8; Thu, 27 Sep 2007 09:22:53 +1000
EHLO localhost
250-localhost Hello localhost [127.0.0.1] (may be forged), pleased to meet you
250-ENHANCEDSTATUSCODES
250-PIPELINING
250-8BITMIME
250-SIZE
250-DSN
250-ETRN
250-AUTH DIGEST-MD5 CRAM-MD5 LOGIN PLAIN
250-DELIVERBY
250 HELP
quit
221 2.0.0 my.mail.server.com closing connection
Connection closed by foreign host.

The AUTH line shows what SMTP authentication methods are available to the client. In sendmail, this is controlled by the following lines in sendmail.mc:

TRUST_AUTH_MECH(`EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl
define(`confAUTH_MECHANISMS', `EXTERNAL DIGEST-MD5 CRAM-MD5 LOGIN PLAIN')dnl

By default, LOGIN and PLAIN authentication methods are disabled using the AUTH_OPTIONS line in sendmail.mc. This is where the problem beings - as MD5 auth does not work in Outlook 2007. Outlook will try to use MD5 auth, but fail with an “Authentication Failed” error. This is interesting, as other email clients do not have this issue. To allow Outlook to send mail, we need to enable PLAIN and LOGIN authenticaion. We do this by changing the following line in sendmail.mc from this:

define(`confAUTH_OPTIONS', `A p')dnl

to this:

define(`confAUTH_OPTIONS', `A')dnl

After rebuilding sendmail.cf (by running “make sendmail.cf”), you can restart sendmail to enable PLAIN and LOGIN authentication methods.

NOTE: I think it is a really bad idea to have to do this, as LOGIN and PLAIN authentication methods have NO SECURITY at all. When you send email using this, your authentication details are transmitted in plain text over the internet. This is very bad. Sadly, if you are forced to use SMTP auth and Outlook 2007, it seems that you must put up with this. Even worse is that there is no indication that Microsoft is aware of this problem at all, meaning that my efforts in getting this recognised as an issue by Microsoft will be fairly fruitless.