The benefits and weaknesses of SPF

[Elkeeed]

SPF doesnt allow gateways to pass on mail so when you have a record setup your mail gets blocked by many systems. It was a good idea but unfortunately causes more problems that it stops.

Also on what people are saying about moving ISPs - actually it is pretty sensible to blacklist all residential IPs because the main source of spam is trojan'd zombies sending directly to mail servers. Theres no real legitimate reason to do this since businesses will be using someone like messagelabs anyway and for individuals there is not much benefit to not using the relay provided by your ISP.

[MONK]

www.openspf.org will also check your domain, you can also check others domains as well to see how they are setup.

There are a huge amount of small businesses that send/receive mail via there broadband connections but a lot of them are used to getting them selves unlisted in firewall rules or getting the clients to do it. For the time being it is the price you pay for running your own server, but I have never had a big problem (maybe two delists) and have been doing the same for a very long time.

[Afty]

It was a good idea but unfortunately causes more problems that it stops.Not so sure about this myself. SPF works extremely well except in a few corner cases - and in those cases, I believe it's well worth the burden of having a few mail admins reconfigure their setups, in order to save orders of magnitude of spam.

I've used SPF for some time with varying degrees of strictness, and it makes a heck of a dent in span received.

[IanCumbers]

SPF doesnt allow gateways to pass on mail so when you have a record setup your mail gets blocked by many systems. It was a good idea but unfortunately causes more problems that it stops.

Not sure I understand what you mean. By Gateway, do you mean a relay? You can add multiple 'allowed' senders for a Domain to an SPF record, whether those relays are in the Domain or not.


Also on what people are saying about moving ISPs - actually it is pretty sensible to blacklist all residential IPs because the main source of spam is trojan'd zombies sending directly to mail servers. Theres no real legitimate reason to do this since businesses will be using someone like messagelabs anyway and for individuals there is not much benefit to not using the relay provided by your ISP.

Big companies can use 3rd parties like Messagelabs, but gazzillions of emails are sent from smaller companies using cheap ADSL pipes and I'd bet that ISPs won't give any kind of granular breakdown on the types of 'user' for each of the IPs that are allocated to them. However, as you say, sending outbound via the ISPs mail server should do the trick. If they are a 'good' ISP, they'll have a reliable mail relay. If they are a 'cheap' ISP, then they won't (uhoh - we're back on that argument again - lol)

[Elkeeed]

This lists the main reasons http://david.woodhou.se/why-not-spf.html

If you don't know how email works then it is hard to explain what I mean by gateways. Basically I am talking about gateways at the receiving end not at your end so no you can not just add them to your record. Any company that is running its own mail server on an adsl line doesn't know what they are doing, it simply isn't reliable enough.

Basically SPF is not really good for either of the things it is supposed to do. For anti forgery you should use something like PGP or similar and for anti-spam this method simply doesn't work.

[kandy]

Any company that is running its own mail server on an adsl line doesn't know what they are doing, it simply isn't reliable enough.


How do you think Small Businesses work then Elkeed, do you think they all have 155Mbit/s pipes to their premises =)

[Afty]

If you don't know how email works then it is hard to explain what I mean by gateways.Eh? He seems to understand how email works just fine, and was asking you to clarify a term you used somewhat ambiguously...
Basically I am talking about gateways at the receiving endI understand what type of mail setup you're talking about, but I believe you're incorrect:

If your inbound gateway is on the edge of your SMTP network, it should be setup to use SPF. Other inbound gateways nested further inside your network should either *NOT* use SPF, or should have the "friendly" edge gateway as a trusted host (meaning they don't do any filtering for mail from it).

Basically, the only time SPF creates an issue for mail networks with multiple inbound gateways is when they are incorrectly configured.

The benefits of SPF are immense (has really cut down on the amount of spam I receive, and all the companies I've worked with over the last few years) and the drawbacks are minor, very unusual, and usually have simple workarounds (configuring gateways correctly for example).

Any company that is running its own mail server on an adsl line doesn't know what they are doing, it simply isn't reliable enough.Do you work in IT? If so, do you work in the real world or some imaginary world. My employer is an *IT PROVIDER*, we provide IT solutions and support to hundreds of small and medium businesses. Not only do many of them run their mail servers on a DSL line, but we run our own mail server on a DSL line.

Of course, we have a secondary backup server elsewhere, but the solution of running the mail server in-office on the DSL line is a much better fit for our needs than anything else by a long margin. This is also the case for scores and scores of our customers.

Basically SPF ... and for anti-spam this method simply doesn't work. WTF? What experience are you basing this on? How many mail servers have you admistered over the last few years?

SPF has put an enormous dent in the amount of spam I've had to deal with from a variety of mail servers (OTOH 5 different servers in the last few years, and many many hundreds of domains, thousands of users) and I couldn't even begin to tell you how many spam mails have been caught by our SPF checking... I suspect it ran into the millions in the first few weeks of operation though.


Really, what are you basing all this anti-spf diatribe on?

[Elkeeed]

I don't have to answer to you. I am an IT admin with far more experience in this area. Not only have I admin'd for both corporates and small business but have also written mail server software so I do know what I am talking about.

It wouldn't take you much googling to check what I am saying and find it to be true but if you want to ignore my experience and find out the hard way then that is your choice. The link I gave above isn't the best summary, it's merely the first that came up in a quick search for summarys. There are various pages which go into it in a lot more detail if you would like to improve your knowledge rather than arguing for the sake of your ego.

Any small business running its own server looks pretty unprofessional when its customers get bounce messages any time the connection or power goes down. If you run your own server because you want to use something like exchange then it makes sense to pass it through a reliable relay service. The TCO of doing it this way is far less and its the way most small businesses that I have dealt with work.

Afty: your whole section on gateways shows the naive reasoning that brought around the problems with SPF in the first place. Yes there is a way that email can be setup for it to work fine. However in the real world there are external influences that mean it can't and isn't always set up in this manner. There are many configurations where mail goes through a non-trusted gateway or one which you do not have direct authority over and that is a perfectly legal setup. So no the answer is not to enforce people to rework their entire setup in order to work with a badly cobbled together system, its to use one that actually works.

[kandy]

Any small business running its own server looks pretty unprofessional when its customers get bounce messages any time the connection or power goes down.

Elkeeed for someone that has such a strong opinion on SPF (which is DNS based), and that has written mail server software I would be gobsmacked if you knew nothing about MX records, specifically the priorities you can set.

[Afty]

I don't have to answer to you.Indeed you don't - but if you want people to take your views seriously you might want to consider addressing the points I make instead of making posts mostly filled with ad hominem or very non specific descriptions of your "vast" experience.

I'm finding it really hard to reconcile your claimed level of experience with your responses in this thread.

I am an IT admin with far more experience in this area.Really? Far more experience than who, may I ask?

It wouldn't take you much googling to check what I am saying and find it to be trueI can google for articles to support just about any set of ideas... within a few clicks I can find dozens of articles supporting intelligent design, evolution, and some about a flying spaghettie monster.
That doesn't make any of those ideas more "true" than any of the others.

I find it quite amusing that you suggest I should "Google for some articles" and that somehow that will override the years of practical experience I have in the field of administering mail servers for scores of companies.

Any small business running its own server looks pretty unprofessional when its customers get bounce messages any time the connection or power goes down.Eh? I assume, having written mail server software, that you can understand what the following does:
company.co.uk IN MX 10 office.company.co.uk.
company.co.uk IN MX 20 backup.company.co.uk.

For those who don't write MTAs in their free time, it means that emails should be sent to the office normally, but if for some reason it can't go there, it will go to a backup server instead.

However, there are MANY companies out there with a budget that doesn't stretch to the provisioning and administration of a secondary mail server - for them the DSL solution works just fine most of the time. The rest of time, they're not spending money on spare machines, so that's fine too.

So no the answer is not to enforce people to rework their entire setup in order to work with a badly cobbled together system, its to use one that actually works.No-one is forcing anyone to rework their entire setup - the beauty of SPF is that it's optional. Organisations can CHOOSE when they want to leave the dark ages behind.

As for "using one that actually works" would you care to suggest a drop in replacement for SPF? And also care to tell us why this amazing solution doesn't already have a rapid adoption rate among MTAs around the world?

Could it be that there doesn't exist a magic alternative, and that SPF is making the best of a bad situation?

[RTO]

The discussion in this thread has spun off from this thread (http://forums.multiplay.co.uk/showthread.php?t=55138).

[Freelance]

E4ovvsUG-KY

[MONK]

Any small business running its own server looks pretty unprofessional when its customers get bounce messages any time the connection or power goes down. If you run your own server because you want to use something like exchange then it makes sense to pass it through a reliable relay service. The TCO of doing it this way is far less and its the way most small businesses that I have dealt with work.

The fact is most MTAs will wait between two and seven days before dropping mail most lines are not out for that long. I look after about 400 domains of which about 150 or so only point to there DSL line and we have only once had a problem with the connection being out for this length of time (BT wiped out something, had an ETA of 9 days downtime.)

A simple grps connection and an MX record update and it was backup up and running a couple of hours later and there was NO bounced mail. If you get bounced mail from a line being down for a while, then you have a miss configured system!

[MONK]

Theres no real legitimate reason to do this since businesses will be using someone like messagelabs anyway and for individuals there is not much benefit to not using the relay provided by your ISP.

Both messagelabs and frontbridge advise using an SPF record on your domain as well as using there service e.g. including spf.frontbridge.com.

I personally wouldn't use my ISP to relay as they tend to get targeted by spam (as they are an open relay for domestic customers.)
You seem to end up with more hits of being a spam source than not.

[Elkeeed]

Eh? I assume, having written mail server software, that you can understand what the following does:
company.co.uk IN MX 10 office.company.co.uk.
company.co.uk IN MX 20 backup.company.co.uk.

But as you say most small businesses don't have the budget to run 2 servers at different sites. This is why I am suggesting that using a company that uses offers a managed server is a better solution because they can pass on economies of scale.

You have to appreciate for me this is a very frustrating thread because we are talking about intricacies in the way mail works. You obviously know how it basically works which is why I sound vague but I am just trying to put it in simple terms. I don't really want to end up explaining all the advanced problems with smtp that add up to the reason. I'm asking you to trust me or look into it further for your own benefit.

No-one is forcing anyone to rework their entire setup - the beauty of SPF is that it's optional. Organisations can CHOOSE when they want to leave the dark ages behind.

Mmm, the problem with that is that this is happening at their end and your mail is getting silently dropped. They don't know they are dropping your mail because they see you as spam.

As for "using one that actually works" would you care to suggest a drop in replacement for SPF? And also care to tell us why this amazing solution doesn't already have a rapid adoption rate among MTAs around the world?

Could it be that there doesn't exist a magic alternative, and that SPF is making the best of a bad situation?

Unfortunately there isn't one or it would be snapped up. Maybe I should have said 'making one that actually works' to be clearer. SPF won't stop spam because spammers can just create their own SPF records and this will become the case more and more as SPF becomes more popular. If SPF wasn't making the situation worse then yes I would go ahead and use it for the short term benefit.

Both messagelabs and frontbridge advise using an SPF record on your domain

I'd like to see what you are basing that on because I was under the impression that they only supported it due to demand and I have personally been recommended by messagelabs not to use for the reasons mentioned.

[KingDaveRa]

SPF seems like a good idea to me.

IIRC, most mail gateways will try to send once, then wait for an hour, then wait for 2 hours, and keep doubling the timeouts up to something like 2 weeks, before failing and sending an NDR.

Some MTAs, however, tend to give up sooner. Exchange can be a swine to work out where holes in logic lie, because some messages can dissappear into black holes between MTAs, neither one acknowledging its existence until it bounces.

Thing is, A/S should be taken as a multi-layered thing, using various different technologies to eventually weed out the spam.

For example, at work we use IP lists to ignore known spammers. Next comes the rules courtesy of Spamassassin. This catches a fair deal, along with Sophos using Libsavi to catch some more. Mail is also checked using a sender verify, to ensure the sender email account exists. Next it goes to the user, who will be usually be using some form of Bayesian filter locally to pick up what's left.

That methodology works very well for a medium-sized University; it might not work so well for a small business, as the overheads of running that are huge. Exchange has its own spam system, which for many is sufficient to do most of the work. We also have Puremessage (http://www.sophos.com/products/enterprise/email/security-and-control/microsoft-exchange/), but we're yet to use that in anger.

My point is, there's no one-size fits all method of catching spam. It has to be tailored. ISPs can get rid of the spam produced by the lazy spammers (most of them), but the rest must be taken care of by the user.

This is all, in my HO, btw :)

[RoboStac]

Unfortunately there isn't one or it would be snapped up. Maybe I should have said 'making one that actually works' to be clearer. SPF won't stop spam because spammers can just create their own SPF records and this will become the case more and more as SPF becomes more popular. If SPF wasn't making the situation worse then yes I would go ahead and use it for the short term benefit.

They can only create records for domains they control though, and at the moment, a large amount of spam is using faked domains, which is what spf stops. It's not a perfect way to stop all spam, but its a good way of making sure if you get an email from a domain, it was actually sent by them, which seems useful to me.

[Afty]

You obviously know how it basically works which is why I sound vague but I am just trying to put it in simple terms. I don't really want to end up explaining all the advanced problems with smtp that add up to the reason. I'm asking you to trust me or look into it further for your own benefit.Do you have to take lessons to become that condescending, or is it a birth trait?

What makes you think I haven't looked into it more, or sat in far more meetings with client directors and presented the cases for and against SPF than you? Tell you what, if you really believe you can convince me that SPF is bad for my customers, why don't you actually POST the reasons?

I can tell you now that you will be barking up the wrong tree... the decision to implement SPF in almost cases I've known has been a decision driven by a business need. You seem obsessed with technical hangups and ramifications - but if you work for anyone other than the government or a research lab you have these things call business cases... and they are far more important than any one minor technical issue.

I am not claiming that SPF is technically perfect, nor am I claiming that it does not have any drawbacks - I *am* saying that as the state of SMTP and internet mail relaying lies right now (and for the last few years) SPF is a sensible and effective measure deployed to meet business needs like reduced spam, massive reduction in volume of mail requiring more computationally expensive authenticity evaluations, reduction in joe jobbing, lack of susceptibility to forged emails - which has been an issue on a number of occasions, especially when company directors leave a board acrimoniously.

Again, you seem to be oblivious to the choices facing companies in the real world. Having a DSL line down for a few hours is simply NOT a major issue for anyone for the reasons Monk has already pointed out.

Further, having a secondary mail server hosted in a datacentre is far FAR cheaper than having your primary mail server with all the associated giga/tera bytes of storage also sat in said £££ per half rack/per amp datacentre. A secondary server that can cache a few thousand emails for an hour or two while a DSL or similar line comes back up is ORDER OF MAGNITUDE cheaper than a primary mail server.
Furthermore, in a setup with a primary mail server hosted in the datacentre, your employees can no longer send/receive mail within the office (inter departmental mail is usually a high priority than outbound/inbound mail for customers or suppliers) unless you run ANOTHER large storage-heavy server within your office (or offices via VPN or other bridge) - which adds yet MORE expense.

For most companies, 99.9% is plenty more than good enough for a service like email which is not a primary business function - the same goes for level of delivery, and accuracy on spam. This can easily be achieved with a DSL line for primary email transit and web access, and a rollover dialup/ISDN capability (many companies don't even opt for the rollover).
There are companies out there who need several more 9s than that.. but they are rare, particularly in the SME sector where budgets are tight.

Seriously, what EXACTLY is your experience, how old are you, how many companies have you managed email for, and for how many years? Do you run your own email server now, and what exactly have you written, where was it deployed and who runs it? All your posts on this forum sound like someone who has "done the reading" but not had much real world experience.

Could you answer just a select few of those questions so that we can actually get an idea of WHY you seem qualified to debate the merits of SPF on this thread with a bunch of people who have SIGNIFICANT real world experience of managing email for many, many companies?

[Elkeeed]

I can't talk about who I worked for or what specifically I have written that is directly relevant. I know that sounds very mysterious but you will have to take it or leave it. Suffice to say I wrote a mail server from scratch with various additional features, mainly to improve security/resilience. In order to do that I had to scrutinise the relevant RFCs and white papers, 3rd party mail server code go over policies of the major ISPs that do nonstandard things with their mail and go through various unusual but surprisingly common mail setups. Based on the fact that most people use standard mail servers rather than writing their own very few people in the world will have spent a year or more on a project like that, so I think I am pretty safe to assume they won't be on this forum.

Plus if you want to talk about condecending, it was you who brushed aside my reasons for not using SPF without attempting to answer any of the concerns.

With the mail server running on adsl issue, you seem to be arguing with yourself over that as you've brought up several issues that I didn't even mention. So to save confusion lets drop that and concentrate on SPF.

OK, you don't like to look at it from a technical standpoint lets look at from a business pov. Having SPF records will cause more of your mail to fail to reach its destination than not having them. If you deal with a small number of people then its all very well to workaround issues you have with specific people but if you deal with a vast number of different people then that is not really viable. Its that simple.

To turn it around and talk about filtering received mail, a large enough proportion of people have setup SPF incorrectly either meaning you will filter their legitimate mail or let illegitimate mail pass anyway.

[Jester]

Afty, now come on stop applying logic and sense, it's Chirstmas ... get drunk and spout rubbish.

[Afty]

I can't talk about who I worked for or what specifically I have written ... I know that sounds very mysterious but you will have to take it or leave it. Suffice to say I wrote a mail serverThis is getting more ridiculous by the second. You wrote a mail server on your own, "from scratch" (why on EARTH would you write a mail server from scratch with the wealth of materials available out there to use... university project perchance?) in a year (or thereabouts) - and you cannot tell us why or for who?

Things like Qmail and Postfix etc. have orders of magnitude more man-hours than that invested, and you were writing yours to provide *MORE* security and resilience than they could provide?

Your credibility is looking very tenuous.

Having SPF records will cause more of your mail to fail to reach its destination than not having them. You have written this as an absolute, but it is definitely NOT true in all scenarios.
ISPs are using SPF not just as an absolute "drop it your pass it", but in order to assign weight to spam filtering.
for example:
SPF record present and DOES NOT MATCH : Massive Negative Weighting
SPF record present and DOES MATCH : Significant Positive Weighting
SPF record not present : Small Negative Weighting
SPF present, does not match, but server is on local subnet to valid IP : Small Negative Weighting

This can go on forever with a whole host of conditions and weightings. It is important to note that many mail service providers who use SPF for spam weighting will assign a negative modifier/weighting to your mail if you do not have an SPF record. If your mail is already on the borderline due to content/volume from domain etc. this can result in your perfectly valid mail being flagged as spam and not delivered because you *did not* have an SPF record. I've dealt with this personally a few times over the years.

In all my years I have yet to deal with a single mail dropped due to the presence of a well configured SPF record...

a large enough proportion of people have setup SPF incorrectly Shock horror, misconfigurations result in undelivered mail, who would have thought? We actually deal with this scenario sometimes, and it usually involves a phone call to us asking about it, and a phone call from us to the originating mailer, after which they can sort out their reconfiguration.


It's really, *REALLY* simple, the business benefits of SPF far outweight the minor negative impact of a few support calls and a few dropped mails (which in the real world almost NEVER happens - the percentage of companies with a setup which is problematic for SPF is very VERY low).

[kandy]

Having SPF records will cause more of your mail to fail to reach its destination than not having them. If you deal with a small number of people then its all very well to workaround issues you have with specific people but if you deal with a vast number of different people then that is not really viable. Its that simple.

To turn it around and talk about filtering received mail, a large enough proportion of people have setup SPF incorrectly either meaning you will filter their legitimate mail or let illegitimate mail pass anyway.

Quite a good point as the major problems I have come across with spam filtering using combinations of DNS based RBL's and SPF, are that your legitimate customers have incompetent IT staff who have done stupid things like forgetting to setup matching forward reverse DNS, have their server identify itself as "EXCHANGE01.LULZ.LOCAL" or broken SPF records.

While we always point out the true reasons for failed mail, I can assure you that an MD couldn't give a **** about misconfigured DNS at their end when they are awaiting an important proposal.

Hence the flaw in your argument is, we implement whitelists on all of our mailservers that we deploy to combat this.

Spelling etc. may be out the window on the above as I've taken Jesters advice and consumed a large quantity of a rather nice red wine =)

[Jester]

And after the large quantity of good red wine , a vast quantity of medicore red wine maybe consumed.

[kandy]

And after the large quantity of good red wine , a vast quantity of medicore red wine maybe consumed.

Its on the way...... get it down you and join some double vision JOPS :p

[Elkeeed]

Afty, all you have done in this thread is question my credibility and make fun rather than put 2 and 2 together and make an educated guess about why I might not be able to talk about it. Obviously, no I didn't do all the work myself and no my Uni project was a video editing program. To be perfectly honest you seem to me to be an obnoxious bafoon and I have no further desire to continue this conversation with you. Feel free to wallow in your own ignorance.

Kandy you seem to be much more sensible but this thread is becoming more hassle that its worth so I'm stepping out. But I will leave you with this, I understand what you are saying, whitelists are good if you have regular correspondants, but if you have many emails coming in each from different individuals then that method becomes too much of a burden. And since you don't control what they are doing on the receiving end I would much sooner filter by SPF than publish my own records.

[kandy]

I understand what you are saying, whitelists are good if you have regular correspondants, if you have many emails coming in each from a different individual that method becomes too much of a burden.

Its fairly easy to conjure up a script that grabs the addresses of outbound recipients you send to that auto populates your whitelist. Takes away alot of the burden.

[Jester]

I knocked up an email server in 1988. Reason why - there werent any really then. Security considerations - none, 2 pc's over serial, if my brother spammed I could always walk to his room and instigate 'at source' anti-spam procedures.

Let the wine flow free. Note - wine rarely contains spam. (if it does, best take it back)

[Cabe]

Spam in wine is called spillage. The carpet has no way to send a source quench message other than to spread it over a vast area and hope the sender remains observant.

[Elkeeed]

<snip>

[MONK]

I'd like to see what you are basing that on because I was under the impression that they only supported it due to demand and I have personally been recommended by messagelabs not to use for the reasons mentioned.

Messageslab was very much pro SPF and had a few white papers on the subject with good results (remember it just helps cut spoofing in the end people can still spam you with a legitimate domain.) Then all of a sudden the white paper links were removed and no mention of it was made.... Talking with the guys at messagelabs they were just told to shut-up about it one day, but in person (or over the phone) they still advise it.

Frontbridge and postini (google) both now filter by SPF (interestingly Postini pre google didn't used to but it seems have started doing so.)

TBH it's never taken me more than a few mins to setup and maintain e.g. include:spf.postini.com + static IP of customer.


Anyway drinking time! :D