Andy Hedges’ Tech Blog

Paul Hammant recently blogged about the DYST method for reducing spam. I’m skeptical of any system which on its own proffers to reduced spam considerably. In the case of spam like everything else in IT there is no silver bullet.

DYST works the following way, according to Paul’s post:

1. I create an email from on my desktop and ask my mail server to deliver it paulhammant.com.
2. The hedges.net server calculates digest using a well know algorithm (presumable using a cryptographic hash) of the mail
3. hedges.net then relays the message (but not the digest) to paulhammant.com
4. paulhammant.com then calculates the digest.
5. all connections are terminated
6. paulhammant.com waits an arbitary amount of time (1 hour is suggested)
7. paulhammant.com then looks up the MX of the hedges.net server in DNS and asks if the mail with the earlier calculated digest was sent by it (hedges.net) to paulhammant.com. If yes then the mail is delivered to the end user if no then presumable it is piped to /dev/null or similar.

He also states the SPF is broken and DYST is better. SPF works briefly this way:

1. I send a mail using my server hedges.net to paulhammant.com.
2. My server opens a connection to paulhammant.com and send the message down the wire
3. paulhammant.com sees the from header is from a hedges.net email address and looks up using DNS the SPF record. This record stores the ip address(es) or hostnames of authorised servers to deliver mail for hedges.net
4. paulhammant.com then compares the address of the server connected to it to see if it matches these credentials and then takes a decision on whether to deliver.

Now DYST seems remarkable similar to SPF in that it is a call back check, but with even more over head. Paul points to this article detailing the ‘problems’ of SPF. Now I don’t agree with this article. It makes the assumption that SPF is your only method of spam detection. This is rubbish SPF should be used as part of a spam scoring mechanism. For example I know that apache.org publish SPF and I know that I want email from Apache so if an email is from apache and a server in its SPF then I’m going to trust it (I have spamassassin configured this way). Now DYST actually suffers from all the issues of SPF and more.

Problems it shares with SPF

• To be truely effective in its own right every SMTP server on the ‘net needs to support it. Paul does address this by saying that a gradual phased migration (of every SMTP server on the net) would have to take place; the same argument applies for SPF.
• There a numerous issues around mail forwarding, routing and relaying which apply to both methods

Problems it adds in addition

• I have to wait 1 hour for my email! This will be a show stopper for many.
• Managing digests puts additional strain on infrastructure

Problems it solves that SPF doesn’t

• I was thinking that it solved the problem of an SPF recorded IP address being compromised but it doesn’t. If the IP address is compromised both methods are helpless.

Other players

Also of note in this vain is Domain Key. Both SPF and Domain Key have failed to reduce spam in the 4 years or so they have been around.

Summary

Therefore with this in mind I’m failing to see why this is a revolutionary spam busting technique.

I may have missed the point terribly, I often do. Paul is a bright guy, I have been reading his blog long enough to realise that - Dion seems to agree with Paul too.

---

update: 13-Nov-2006

Paul has written an excellent reply to my post, sometime ago actually, but I just noticed it. Paul raises so good counters to my objections. However I would like to provide some further clarification:

Digest Expense

I wasn’t refering to the overhead of computing cryptograhic hashes, I was actually refering to implementing a data store, index and retrival system for these identifiers. If every message you send requires you to respond to a query on its id at some arbitary time in the next hour you have to have a way of doing it. This requires: software to perform the lookup, so all SMTP servers must implement this and it also requires software to serve the lookup; more development. On top of that we have the expense of making the lookups once the software is built.

One hour wait

I had a assumed it would confirm the id 1 hour after the message was recieved. I was wrong. It seems that the server would store the if for 1 hour before discarding it. I would suggest 1 hour is a little too shorter time, especially if mail is being stored and forwarded, as is sometimes the case.

Not different to SPF?

SPF does not require any enchancement to DNS software or the specification of DNS. SPF uses already existing types within DNS, namely the TXT record type. Libraries exist in most modern langauges to query DNS. It is actually DYST that requires huge amounts of development server side.

One more thing

Having these id lookups introduces statefulness to the SMTP specification, this will lead to difficulties with multiserver environments. Let for a minute consider gmail. A quick DNS dig reveals:


gmail.com. 3568 IN MX 50 gsmtp183.google.com.
gmail.com. 3568 IN MX 5 gmail-smtp-in.l.google.com.
gmail.com. 3568 IN MX 10 alt1.gmail-smtp-in.l.google.com.
gmail.com. 3568 IN MX 10 alt2.gmail-smtp-in.l.google.com.
gmail.com. 3568 IN MX 50 gsmtp163.google.com.



these are the servers that will accept mail for gmail. So which one do we perform the DYST against? If we just pick one at random then these servers must replicate their ID databases, otherwise if we must know the right server to ask we need a way of specifying the ID database server. Note that it is entirely possible that none of these servers actually sent the mail and so trying to reconcile that way wouldn’t work, it is also possible that each one of these addresses facades multiple servers.

Just for fun I did an SPF lookup on gmail and it looks like they support SPF, using a normal SMTP and DNS:


gmail.com. 300 IN TXT "v=spf1 redirect=_spf.google.com"


This entry was posted on Thursday, August 31st, 2006 at 11:06 am and is filed under Spam. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.