Finally, after months of sketchy integration test failures and banging heads against keyboards, last Friday I figured out the issue with our e-mail based integration tests. Read on!
After a number of months of dealing with inconsistent integration test results, I finally took the time to investigate what was happening. Our issue revolved around tests we had written that asserted we were sending appropriate emails in response to server events. To do this, we use the netDumbster library, based off the Dumbster java library. Both provide a localhost SMTP server of sorts which you can then assert against – for example:
1 2 3 4 5 6 7 8 9 10 11 12 13
With netDumbster, you can test your sent email code without having to write some complicated mock or test service that caches sent email to disk. It solves a pretty important testing issue that a lot of people spend time solving themselves, or even worse, avoid completely.
But, despite this excellent testing tool, our email-based integration tests were failing at a pretty consistent rate. When we re-ran those tests, they would often pass the 2nd time or when run individually. After a few hours of debugging which included downloading the netdumbster source code, I eventually concluded that it was an issue with SmtpClient itself, a part of the system.net.mail namespace included with .NET.
Sure enough, some googling found reports from other developers indicating that there was an issue with SmtpClient not sending a proper “QUIT” signal when it disconnects from a SMTP Server, and also had issues releasing the socket it uses. It also says the bug was fixed in .NET 4.0, which is the version that we’re using, so that didn’t explain the issue. But, digging through the comments, I noticed one thing:
To save you further lookup… the SMTP QUIT command is sent upon disposal of the SmtpClient object.
Sure enough, we weren’t calling Dispose on the SmtpClient object. I added that in a try/catch/finally block and it fixed our issue! All our integration tests run consistently now. Keep this in mind if you’re ever playing with email testing.