Is this code available anywhere?
No; it's written in Arc, which is itself not released yet.
Is there a Bayesian filter for Outlook?
I know of nine so far: Spammunition,
Is there anything that can protect my company's server?
The best commercial server-level Bayesian filter is probably
one of the best open-source Bayesian filters, can also be run
on the server.
Most commercial server-level spam filters are still rule-based.
But there are starting to be some
that use Bayesian filtering. The way to find them is probably
to search in Google.
The question to ask the salesman is, does the filter learn to recognize
spam based on the spam and nonspam mail we receive? If it
doesn't learn, it isn't Bayesian.
Does Arc/Lisp have some advantage for writing this kind of software?
Lisp's symbol type is useful in manipulating databases of words,
because it lets you test for equality by just comparing pointers.
(If you find yourself using checksums to identify words, it's symbols
you mean to be using.) Prolog, Erlang, and Ruby also support symbols.
So does Python, to a degree, though I don't think it has any syntax for them yet.
More generally, Lisp was designed for rapid
prototyping, and this application involved a lot of that. I probably
spent 95% of the development time typing expressions into the
toplevel, trying variously tweaked algorithms on individual emails.
Do you mind if I write filters based on this algorithm?
Of course not. I don't claim to have invented anything new here. Bayesian
text classification is an old and established field. If there is anything new
in this article (at least, it was news to me) it is that such a simple
variant of it is so effective at filtering spam.
So by all means go write spam filters. It is a very rewarding hack. If
you end up creating something other people can use, let me know and
I'll make a link to it.
I don't know Lisp; can you explain the algorithm to me?
It's expressed in math notation in Hackers
Could you use Bayesian filtering to make Web content filters?
I've never tried it, but yes, I think it would work well.
Will this algorithm filter out 99.5% of my spam with no false positives?
It does filter out 99.5% of my spam. I got similar results for the one
other spam collection I've tested it on so far. (I couldn't measure false
positives, because I only had the spams.)
I do worry a bit that my email might just be especially easy to filter.
A lot of my spam comes from "opt-in" spammers like Qves and Virtumundo,
and that stuff is really easy to detect. Plus my own mail is full of references
to programming that make it easy to distinguish from spam (though not
"programming" itself, ironically, which often occurs in spams selling
So if you do try implementing this filter for yourself, I'd appreciate it if you
could let me know how well it works for you. My gut feel is that it will work
pretty effectively for almost everyone, but it would be reassuring to hear
Could spammers fool Bayesian filters by filling their spams with random words?
They would have to get rid of the bad words as well as adding neutral ones.
Only the most interesting fifteen words contribute to the probability,
and neutral words like "onion", no matter how many there are of them,
can't compete with the incriminating "viagra" for statistical significance.
To outweigh incriminating words, the spammers would need to dilute
their emails with especially innocent words, i.e. those that are not merely
neutral but occur disproportionately often in the user's legitimate email.
But these words (e.g. the nicknames of one's friends, terms one uses
in one's work) are different for each recipient, and the spammers have
no way of figuring out what they are.
Once this software was available, couldn't spammers just tune their spams
to get through it?
They couldn't necessarily tune their
emails and still say what they wanted to say. If they wanted to send you to a url that
is known to the filters, for example, they would find it hard to
tune their way around that.
Second, tune using what? Each user's filters will be different, and the innocent
words will vary especially. At most, spammers will be able to dilute
their mails with merely neutral words, and those will not tend to be much use
because they won't be among the fifteen most interesting.
If the spammers did try to get most of the incriminating words
out of their messages, they
would all have to use different euphemisms, because if they
all started saying "adolescents" instead of "teens", then "adolescents"
would start to have a high spam probability.
Finally, even if spammers worked to get all the incriminating words out of the
message body, that wouldn't be enough, because in a typical spam a
lot of the incriminating words are in the headers.
What if spammers sent their messages as images?
They already do sometimes, and we are able to catch them.
Such emails include a lot of damning content, actually. The headers,
to start with, will be as bad as ever. And remember that we scan
html as well as text. Within the message body there
will probably be a link as well as the image,
both containing urls, which would probably score high. "Href" and "img"
themselves both have spam probabilities approaching pornographic words.
Within the url, the image has to have some kind of name, and these are
usually far from random.
Can your program deal with the spam trick of inserting html comments
in the middle of words?
Yes, I ignore html comments down at the
level of scanning tokens, not even considering them
Spammers sometimes use randomly generated tags to break
up tokens, since html rendering software will usually ignore
meaningless tags. I do allow these to separate tokens, and it
works fine. Broken bits of words simply get high spam probabilities.
Would this article itself be filtered out by your filters?
No. Someone sent it to me as a test, and it wasn't.
Although the article contains a lot of
spam words, it also contains a lot of even more provably innocent
words (like "spammers" ironically, which occurs in a lot of my
real email, and never in my spam). Since only the most
interesting words count, the innocent words
crowded out the spam words in the rankings, and this mail
ended up with the minimum possible probability.
That makes sense, because an article someone writes is more likely
to resemble the content of their own email than it would a spam, even
if the article happens to be about spam.
How well can your software filter out mail that's about spam?
This is a problem for anyone who works on spam filtering. In the extreme
case, if someone you've never heard from merely forwards you a spam
with no additional comment, it's going to be hard to filter out. There it will
all come down to the headers.
But after all, if someone could forward you spam with
impunity, then spammers could forward you spam too. So for
these situations we may have to have a special password
that people could include in mail to get it by the filters, and
also (I have one already) a special trash folder for metaspam
so that it doesn't contribute to the probabilities.
If there is a good side to this, it's that if we can create filters that work
acceptably well for us, they'll work even better for everyone else.
Will this software filter out good automated email
like order receipts and newsgroup faqs?
Mostly not. That kind of mail may be automated, but
the text usually has a very different character from spam.
The only automated responses that tend to get filtered out are the
ones that contain a lot of additional advertising. In effect, these
emails consist of an automated response with a spam appended to the
end, so it is not surprising filters catch them.
Will the kind of people who would respond to a spam use filters?
I think so. I'm guessing here, but I suspect that people
to respond to a spam will often get email through one of the free services
like Yahoo Mail or Hotmail, or through big ISPs like AOL or Earthlink.
Once word spreads that it is possible to
filter out most spam, they'll be forced to offer effective filters.
If filters catch most spam, won't spammers just send more to compensate?
Spammers already operate at capacity.
How do you feel about blacklists?
As ISPs use them now, they're equivalent to very bad filters.
But the information they supply could be incorporated into Bayesian filtering.
Doesn't Bayes' Rule assume the predictors are independent?
Yes, strictly speaking it is only valid assuming the probabilitiy of e.g.
"sex" occurring in an email is unrelated to the probability of "sexy"
occurring in it. Obviously that is not the case.
On the other hand, there is a long tradition of violating
this requirement. When you do that it's called a "naive Bayesian" algorithm
and in practice it works pretty well, just as in practice (if you stay away
from the edges of precision) it works pretty well to treat floating point
numbers as if they were reals.
Does this algorithm take into account the prior probability of a mail being a spam?
No. It assumes it's equally likely for a mail to be a spam or not. So accuracy
could be slightly improved by taking into account the actual proportion of spam
received by each user. You would have to do this by hour though, because
the proportion varies so much depending on the time of the day that the
overall prior probability is useless as a predictor.
Why do you consider 15 words? What's special about 15?
Actually, in the latest version of the filter I use 20, because
marking tokens by context means the
software now sees more distinct
tokens per mail.
Nearly all spams tend to have that many fairly incriminating words
in them. If you
look at more than that, you'll start to include neutral words,
which are noise at best. But you want to look at as many words as you can,
because legitimate emails may well contain two or three words with high
spam probabilities-- if you used only the top 5 words, for example,
you'd start to get false positives.
Will this work for languages other than English?
As long as they use whitespace to separate tokens, it should.