According to just released research from Michigan based OnlyMyEmail Inc it would appear that Sender ID is ineffective as an anti-spam solution. Despite the high profile, and frankly somewhat aggressive PR campaign by Microsoft, the 60 day statistical analysis certainly suggests that it isn’t the Holy Grail of anti-spam that Mr. Gates would have us believe.
Across a 60 day period, OnlyMyEmail discovered that emails sent by a domain without a published Sender Policy Framework (SPF) record were only slightly less likely to be spam than those sent by domains that publish Sender ID information in their DNS records. But, emails returning a positive match for Sender ID were not significantly more likely to be legitimate compared to those without a published SPF record, which is surprising. Most surprisingly, in my view, was the fact that an email that failed Sender ID verification only had a 91.4% chance of actually being spam. This translates into a false positive rate (FPR) of 8.6% if Sender ID were relied upon to accept or reject messages. When I evaluate anti-spam solutions in a professional capacity, be that for a magazine labs review or with my consulting hat on for a corporate purchase decision, anything that returns a FPR greater than 1% is rejected out of hand (no company can afford to lose 1 out of every 100 business communications because it has been wrongly flagged as spam.) With 1 out of 6 emails passing Sender ID verification also likely to be spam, according to these results, and the spam probability of incoming mail from domains not publishing Sender ID data being 49.5%, non-compliant domains are less likely to be sending spam than those who publish SPF records within their DNS (although only just.)
If you are an administrator who relies upon Sender ID to make decisions on whether or not to accept or reject inbound emails, this is bad news indeed. Crikey, some of the first generation anti-spam solutions could return a vastly superior performance when it comes to accuracy than Sender ID on these results. Worse, administrators who publish non-specific Sender ID/SPF records (in terms of their sending IP addresses) increase the chances that their outbound email will not be delivered by almost 79%. OnlyMyEmail take their criticism of Sender ID even further, claiming that spammers now commonly publish SPF entries for their sending domains which means it is being used as a weapon to increase spam delivery rates. This works because many email administrators have become over-reliant on Sender ID.
I remain unconvinced that Sender ID or SPF is the solution to spam, a conviction that has been reinforced by this research. Not that it really needed any reinforcing over and above the simple fact that pretty much every ‘traditional’ anti-spam filter either on the client or server side that I’ve tested in a real world environment has out-performed it in every meaningful way. The FPR figure being the real bottom line here. If you currently are relying upon a Sender ID or SPF bases solution, at the very least I urge you to consider what I’ve written here, and the OnlyMyEmail research, and do some real world testing of your own. If the solution is costing your company more in potential lost business than the problem is in wasted time, well, you don’t need a management consultant to tell you what to do…
