Email Marketing News and Tips

Best Advice From The Email Marketing Experts

Email Marketing News and Tips header image 2

Spam filter inner workings (Final Part)

July 14th, 2008 · 3 Comments | Print Post Print Post

(Continued from Part II)

Filter accuracy

Filters can be made more accurate, for example, by breaking the content down into smaller samples and running both Fingerprinting and Bayesian filters on each of the smaller segments. Removing spaces, tabs and other insignificant characters can yield better results. Once this level of filtering is done, this increases the accuracy of the filter and reduces the chances of a false positive (or even a false negative). So, if the content is broken down into, for example, 5 chunks with only the most significant characters, the filtering system can then score each of the 5 sections individually and then produce a final score for the overall content. If, for example, 4 out of 5 sections score greater than .5 and one section scores less than or equal to .1, then this email could still be determined to be spam. However, sending this same email through the Bayesian filter alone may have yielded .2-.3 (not spam) due to a poisoning attempt.

HTML Syntax and Lint checkers

Some filters (like Spamassassin) also add HTML checking to determine the validity of the HTML itself. By determining whether an email’s HTML content is valid, this can contribute to the final score on the email. This type of checking both validates the HTML and also helps prevents problems by the recipient. The better the HTML works, the more effective the email will be (and the less likely it will be scored higher).

Spam filter software and maintenance issues
There are many different spam filter softwares available both as open source products such as Spamassassin and Bogofilter as well as commercial products. Any spam filtering system requires maintenance by the end user and the system administrator.

Active vs Passive Filtering systems

How a spam filter is implemented is entirely up to the site needing it. Because spam accounts for a significant portion of email traffic on the Internet, most sites have implemented some level of spam filtering at the gateway to reduce unwanted traffic before the email even enters the mail gateway. This is considered to be active filtering. This means that the user has no control over these filters, yet they work to reduce the load on servers to prevent server overload.

Filters such as Bogofilter and Spamassassin, instead, choose to alter the email by adding a score to the header to guide the viewer. This is called passive filtering. By simply adding a score, this then lets the inbox owner make the determination (or not) of what to do with the email. Some applications, like Thunderbird, have even added support for Spamassassin headers to automatically file the email appropriately.

Conclusion

As an email marketer, understanding this level of technical information helps you better understand what filter technologies are out there and how to produce better content to reduce your content’s chance of being filtered. However, the best way to ensure your emails get through the filters is to have your recipients train their filters to allow your content through and also by asking them to add your From address to their address book.

Tags: Intelligent Marketing · Tips

3 responses so far ↓

  • 1 Spam filter inner workings (Part II) // Jul 14, 2008 at 4:41 pm

    [...] in Part III) addthis_url = [...]

  • 2 Spam Blocker Michael // Jul 19, 2008 at 10:06 am

    What about domain keys used at Google and Yahoo. Does that technique work to block spam?

  • 3 Brian Wright // Jul 23, 2008 at 5:48 pm

    Michael,

    Although, Domain Keys could be used as a way to block email, the idea is that Domain Keys is used to verify an email to the recipient. The way Domain Keys verifies is by the following tests:

    1) DK validates that From line domain claim of where the email originated is, in fact, where it originated.
    2) DK also validates that the email’s body content remained unchanged during transit (that the content hasn’t been altered during the send).

    DK verifies these tests by using a DNS published PGP key on the domain in the From line.

    Test 1 could be used as a method to block email. Test 2 could additionally be used to as a way to block mail. Presently, I am not aware of any sites using Domain Keys, yet, to block email.

    I will write an in-depth blog article describing how Domain Keys works in more detail to better answer your question.

Leave a Comment