Methods and techniques of automated removal

There are several different techniques of automatically scanning an email and making a determination of whether or not that email is considered ’spam’ using software. The following list describes the most common techniques in use today:

1. Keyword matching filters
2. Fingerprint filters
3. Bayesian filters (Part II)
4. Database comparison filters (Part II)
5. Shared database filters (Part II)

Each of these above describes a class of filters. Some sites use combinations of each of these styles of filters to get maximum benefit.

Keyword Matching Filters

This technique is the weakest of all of the spam filtering methodologies. This technique searches for specific words within the body (or header) of the message. The main problem with this technique is that many of the word search methods are case sensitive. So, searching for the word ‘FREE’ is different from searching for the word ‘Free’ which is also different from ‘free’. Even if the searching method can be made case-insensitive, the person wanting to get past this filter type could deliberately misspell or add spaces to the word such as ‘F R E E’ or ‘FRIEE’ or ‘FR EE’. These simple changes to the word prevent the keyword filter from catching this new version.

As a result, a person implementing only keyword matching will constantly have to update and alter each new word incarnation that is found. Thus, instead of spending time deleting the email only, now you have to delete the email and then rework the filter.

Fingerprint filters

As a result of the inordinate amount of time required to manage keyword filters, other filtering methodologies have been created. Fingerprint filters (otherwise known has hash filters) use the email body (and rarely some of the header information) to create a hash value of the most significant content in the email. Usually header information is omitted due to the changeable nature of the headers. The body content is then hashed (condensed) into a 32 or 64 bit character string. This string is usually stored in a database table for future reference.

When a later email arrives and the body hashes to the same value as previously seen content, the email can be marked as ‘been seen before’ (see Active vs Passive Filtering — Part II). Depending on how the filtering system is designed, fingerprint filters can then block, bounce or deliver the email based on if the content has been previously marked.

A note about hashing: Unfortunately, hashes aren’t perfect. Condensing a lot of data down into a 32 or 64 bit character representation leaves the system open to collisions.  A collision is when unlike content hashes to the same hash value. As a result, a collision can then produce a false positive and act on it inappropriately. So, automated fingerprint filters are rarely used alone. Another issue with this technique is that by adding random text (or spaces) to each email sent, the hash value will change even if the most substantial portion of the content remains the same (see Filter Accuracy — Part II).  A different hash value, in turn, makes the system think that it is a new unseen email.

(Continued in Part II)

There are two really cool things about this new feature:

1)    It’s built right into the simple three-step workflow with which you are already familiar.
2)    It works exactly like the Salesforce.com advanced filter feature for reports.

Let’s talk about how you’re going to use this new tool to get the maximum benefit for your email program.

The first thing that you’ll need to do, of course, is to log in to your Salesforce.com account, click on the ‘BlastWizard for AppExchange’ tab, and then click on ‘Get Started’.  Next, select an email message and click on ‘Next’ again.

At this point you’ll be looking at the list building option screen seen here:

The fun begins when you select ‘Create your list by segmenting your leads and contacts’.

From here, you have a number of choices in the dropdown menu from which to choose depending on what you’re looking to achieve:

•    All Leads
•    All Contacts
•    All My Leads
•    All My Contacts
•    Segment All Leads
•    Segment All Contacts
•    Segment My Leads
•    Segment My Contacts
•    Build a List By Searching My Account

Most of the choices are pretty much self-explanatory, but we do want to make a quick note on what we mean by “All Leads” or “All Contacts”.  When we say “All”, we mean all of those leads or contacts that you have access to via your Salesforce.com profile, which may or may not include every lead or contact in your company’s Salesforce.com database.

Of course “My Contacts” and “My Leads” refer to those records in Salesforce.com that are actually assigned to you.

When you choose any of the first four “All…” choices, BlastWizard will pull those records into the system in real time, and they will then be available to send to as a whole (“Send to all”), or you can select which recipients you want from the group (“Send to the selected people only”).

When you choose any of the “Segment…” choices, you’ll find a familiar-looking filtering page like the one below:

The dropdown menus on the left contain all of the searchable fields that Salesforce.com allows for Leads or Contacts (whichever you’ve chosen as your starting point).

The middle dropdown menu contains the operators like “equal to =”, “less than <”, “contains” and so forth.

Finally, the fields on the right side are where you’ll enter the data on which you wish to segment.

If you’ve ever used the advanced filters in Salesforce.com to create a report, the BlastWizard interface will look very familiar.  And if you’ve never had a chance to use them, you’ll find that they are quite user-friendly and rather intuitive.  Of course, if you ever have any questions about this feature or any other part of BlastWizard, don’t hesitate to give us at a call at 1-800-769-4000.  Happy hunting!

What can you do to ensure your emails make it through the filters?

The easiest ways to ensure your emails get through the filters are as follows:

1. Make sure the recipient has opted-in to receive your emails
2. Request the recipient to add your From address to their address book (might not work in all cases)
3. Send out verifications about every 6 months to ensure the recipient still wishes to remain opted-in
4. If you change the From address, re-verify the recipient and request them to add this new address to their address book
5. Have the recipient, if a corporate or government user, contact their IT department and explain they want to receive emails from your company.
6. If you find that a company is using Postini (or a similar filtering service), work with your Email Service Provider (such as Boomerang) to ensure your email sending IP address is allowed to deliver emails.

Doing the above ensures that the recipient really wants to see your content and will make it through the filters. Adding the address to the address book, in many cases, tells the mail server they want to see the email. In some cases, however, the email client does not communicate this address book with the mail server. So, an address book addition may not help in all corporate cases. Adding to an address book helps for sites like Yahoo, Hotmail, Gmail and similar web based mail clients where the address book is integrated with the server.

What about false positives and how does it affect my campaign?

False positives occur when a filter mistakenly traps legitimate email and doesn’t deliver it to the inbox. This happens when a legitimate email triggers a filter and lands up in the quarantine, in the spam folder or, in some cases, deleted. This issue can affect your email campaign more when the content uses certain words or phrases, or by including certain content elements in your HTML email design. Again, another blog entry will go in depth to describe the internals of filter systems that are available today.

As far as percentages caught as false positives, that’s really dependent on the type of filter, how the filters are built, how well the filters are initially trained, how often they are maintained, how often the administrator retrains the filters and where the filters exist. Some spam technologies claim false positive percentages to 0%. Some false positives can be as high as 10-12% (or higher). The goal of this blog article is to describe where the filters sit and specific types of filters that get implemented. The design internals of the filters is planned as a future blog post.

How dedicated IP addresses help.

If you continue to see lower delivery numbers than you expect, it could be because you are not using a dedicated IP address. Dedicated IP addresses allow you, as an email marketer, to limit email traffic to only one (or a few) IP address(es) dedicated to your email traffic alone. Once a dedicated IP address is set up for you, you can have this IP address whitelisted by many different organizations to improve your inbox deliveries to those organizations. Because a dedicated IP address allows you to whitelist, having a whitelist affects Mail Gateway level filters (which are usually the most problematic type of filters).

Additionally, dedicated IP addresses can be named appropriately in DNS and specifically set up for additional features like Yahoo Domain Keys and SPF which also improve deliverability.

Conclusion

Email filters are here to stay and likely to filter more and more as time progresses. This is a critical time to be thinking about how to get your emails to your recipients through these filters. Recipients will need to stay on top of their own mail administration team to ensure even valid emails make it through. Thus, this is really a two sided issue. While you as an email marketer should understand and do the recommended items above, you will also need to help educate and guide your recipients to do the things they need to do to ensure you can continue to contact them.

Types of Filtering Implementations

To explain exactly how filtering systems work in detail requires an explanation beyond the scope of this article. Please see the blog article named Spam Filter Inner Workings.

Corporations and Filtering

Corporate filters are some of the most stringent filters out there next to certain governmental agencies. Corporations do not abide by any set of standards or rules when it comes to filtering. So, corporate filtering implementations vary wildly from business to business. Some businesses offer employees the use of Exchange/Server level filters, but many use Gateway level filters that exclude the email recipient from access.

Many corporations are buying into filtering services such as Postini. With Postini, the corporation has their email routed through Postini’s email servers for filtering which then gets delivered (or not) to the corporation after having been filtered. The issue with handing off corporate filtering to a third party is one of scale and confidentiality. Because services like Postini serve many clients at once, it’s possible that a certain times of the day Postini’s servers could become overloaded and lose some of a company’s email. If the company is a sales oriented business, one lost email could be enough to blow a deal. For this reason, contracting out to shared filtering services may not ultimately work for critical email accounts (even if services like Postini do eliminate unwanted emails).

Also, because the company’s emails are filtering through Postini, this possibly means that prying eyes could be viewing company confidential emails unless Postini signs a non-disclosure agreement. Even still, a non-disclosure doesn’t protect emails from accidentally becoming publicly viewable. It simply allows a company to sue Postini should that eventuality occur.

Blackholes (RBL: Real-time Black List) vs Filters

A black hole isn’t a type of filter even though some people might think it is. A black hole is a type of server level IP blocking mechanism (a kind of dynamic firewall). If an IP is in a black hole list (such as Spamcop), the IP isn’t allowed to deliver any email to the mail server subscribing to that black hole list (as long as the IP remains on the list). Mail administrators subscribe to black hole lists to reduce unwanted traffic to their mail server (just as they do with gateway filters).

Black lists are consulted early in the connection process to allow dropping the connection before any content has been delivered. A filter is different from a black hole because the filter is utilized after the email content has been accepted by the mail server, but before it’s placed into the inbox.

Black hole blocks are usually implemented as a result of excessive numbers of complaints about a specific Email Service Provider or a specific company and usually because the email content is easily recognizable as spam. Reputable Email Service Providers do not allow this type of email to be delivered through their service and, thus, should never have any IP addresses listed on RBL lists.

Grey Lists vs Black Lists

Greylisting is a process that allows mail server administrators to push off an email for later delivery. So, instead of filtering it out (with a gateway filter) or completely blocking it (using a black hole list), it tells the delivering mail server to ‘try again later’. The thinking behind a grey list is that many spamming softwares don’t ever retry sending the emails. So, when these spam softwares are told to retry later, they don’t. Legitimate Email Service Providers like Boomerang’s email marketing platform offer enhanced and extended retry mechanisms to try delivering the emails multiple times over a specified period of time.

Continued in Part III

1. Be on top of your bounce management

2. Email regularly (at least once every 90 days)

3. Reduce complaints by asking for permission, sending relevant email, and not over-emailing

4. Test content/subject lines

5. Make sure your brand is recognizable in your email/from field

6. Maintain your reputation/authentication by using SPF, SenderID and DomainKeys

Thankfully, Boomerang can help you with all of the above!  As experts in email marketing and distribution, Boomerang is in the top tier of ESPs in regards to bounce handling (ask about our ‘Deliverability’ whitepaper!).  Our BlastWizard tool allows A/B subject line testing, and also ‘tests’ the content of your email with our Content Checker, giving you a Spam score before you send your email out.  And, Boomerang can help with authentication.  Let us be your B-to-B email service provider!

There are several levels of filters you will encounter in your email marketing campaigns. The following lists the most common locations or levels of filtering you will encounter:

Locations of Filters

1. Mail gateway filters
2. Server level filters (i.e., Exchange managed)
3. Client level filters (such as Outlook or Thunderbird)

Description of Filters

1. The mail gateway filter is usually the most problematic type of filter for an email recipient. These gateway type of filters are optimized and built by the mail administrator over a site. These filters are usually implemented to help reduce unnecessary traffic to the mail server and reduce the overall processing load. The main issue with this type of filtering is that the owner of the email address has no control over these filters and, in many cases, does not know the filters exist. Usually, the recipient finds these filters exist only after an expected email has gone missing and goes looking for it. If the system administrator has built this system correctly, the filter will retain the missing email in a location usually called a quarantine. The quarantine isn’t always available at all sites, however. So, without a quarantine, the email could simply go missing without a trace.
2. Server level filtering are filters that the email account owner can build themselves directly onto the email server. These types of filters are usually editable through a web based interface. For the email address owner, these are the best filters to use. These types of filters work 24 hours a day / 7 days a week without the client running. Because the recipient is in control, the recipient knows exactly what is being filtered. Thus, the recipient can create new filters to allow wanted emails through.
3. Client level filters are filters that get built in the client using message rules (such as within Outlook or Thunderbird). These filters are typically only installed in one single client. So, the trouble begins when you want to read emails using several different devices or with different computers (i.e., at home, on the web, at work and on your iPhone). With client filters, you have to build the filters for each client or, alternatively, you must leave one email client running 24/7 to support the filters. Even still, these types of filters can be slow to respond (depending on how often Outlook or Thunderbird checks for email). This is the least effective way to filter emails.

Continue to Part II

We’re reminded when speaking to long-time clients who’ve been using the application for years and they ask us about a seemingly obvious (to us) feature that may have come out 6 months ago or longer in a regular BlastWizard update. And while we try to communicate as frequently and broadly as possible about new or enhanced features, we know that we’re not going to reach all of our users.

As such, Boomerang has recently released a Frequently Asked Questions (FAQ) section on the Boomerang Wiki that caters to both the regular BlastWizard user as well as the BlastWizard for AppExchange user.

The BlastWizard FAQ can be found at http://mainwiki.boomerang.com/index.php/FAQ_Index

It is broken into 7 sub-categories for your convenience:

-Getting Started
-Campaign Wizard
-Jobs
-Reports
-Salesforce.com Integration
-My Account
-Unsubscribes

As with all things related to BlastWizard, we are constantly looking to make improvements to this site, and encourage you to submit your own questions that you’d like to see answered as a response to this blog.

Not surprisingly, most businesses are treating email marketing the same way as Clinton - as a last ditch effort to win new clients.  Not a smart move, considering that email marketing has ‘an ROI of $57.25 for every dollar spent, over 150 percent greater than the ROI for non-e-mail online marketing’ (http://www.clickz.com/showPage.html?page=3623879).

Email marketing should be a key part of your marketing strategy - and one that is working concurrently with other marketing tools that you may be utilizing.  I read recently on another blog ‘Adventures in Email Marketing’  (http://www.banane.com/workblog/?cat=10), in which blogger Anna Billstrom discusses email campaigns and contact frequency, that ‘if your inventory can support daily emails, do them.’  Yes, “DAILY”!  Of course, this might not work for most businesses.  But, the point is that email marketing campaigns should be used frequently and should be an intergral part of your overall marketing plan.

But before we go too much further, I want to make a few things clear about Boomerang’s position on CAN-SPAM and spamming in general. As stated in our Acceptable Use Policy and in the Terms of Use agreement, Boomerang does not tolerate spamming. We support the CAN-SPAM law as a bare minimum, and strongly urge our clients to grow their list organically and to not rent or buy email lists, not only because list rentals ruin our reputation and that of our clients, but also because they simply don’t work.

Please also be aware that Boomerang.com is not liable if a client of our’s violates CAN-SPAM. Instead it is the company that is considered the sender (who provided the list, is in the “from” address, and so forth) of the email who is responsible.

As a review, CAN-SPAM covers commercial email, and to be compliant with the law marketers really only have to do a few simple things:

-Include a viable opt-out method, and act on unsubscribe requests within 10 days
-Include a physical postal address of the sender
-Use a real “from line/address” that does not mis-lead the recipient as to who has sent the email
-Make sure that the subject line of the email corresponds to the content of the email and is not mis-leading
-Indicate the email is an advertisement

With all of that out of the way, let’s talk about the call. The 100 page update from the FTC basically clarifies questions that were raised by the DMA and other organizations regarding existing provisions, and ruled on certain changes to the law that had been proposed.

Here’s what you need to know:

1. In the event that you’re co-marketing with one or more affiliates, you’re now able to designate one of the companies as the “sender” of the email. That company should have control of the content, be in charge of the list, be in the “from” line, and be in charge of the opt-outs. This essentially clears up the identity of the sender for the recipients when they receive an email with several advertisers in it. However, the commission did not create a “safe harbor” for the other companies represented in the message, so they are responsible for making sure that the “sender” goes by the book.

2. The update includes several clarifications on what the law considers to be “transactional” email which, as you know, are excluded from the need for CAN-SPAM compliance. However, here at Boomerang we encourage our clients to use our Confirmation Plus tool to add relevant up-sell opportunities to their transactional email, so another part of the update applies. When there is commercial content included in a transactional email, the committee will use the “primary purpose” criteria to determine if it falls under CAN-SPAM. To be on the safe side, Boomerang reccomends that you include the CAN-SPAM compliance elements if there’s any commercial aspect to your email.

3. The definition of a “Physical Postal Address” was further defined, and now an accurately registered PO Box or private mailbox registered with a commercial mail receiving agency (CMRA) is acceptable to the FTC. However the DMA guidelines, which Boomerang supports, still require a true physical address.

4. The next point regards liability as it relates to Forward to a Friend (FTAF). The FTC has said that emails that are forwarded on by the recipient using a FTAF mechanism (that reach the recipient without CAN-SPAM compliance elements) are not considered for prosecution, unless the marketer is paying the recipient to forward the email along.

5. The commission decided to continue the 10 day window for unsubscribes after considering a proposal to reduce that window to 3 days.

6. They also reaffirmed that there is no expiration to an opt-out request, no matter how old.

7. There is a prohibition on charging a subscriber a fee or imposing any other requirements on people who wish to opt-out.

As you can see, for the most part these rulings and clarifications touch on subjects where most responsible email marketers are already doing the right thing. As long as you make sure to include those original CAN-SPAM compliance elements I mention above every time you send a commercial email, you’ll be in good shape.

Upgrading your service plan is simpler

Based on your valuable feedback we modified how ‘Upgrade Service Plan’ works. Now when you upgrade to a higher plan your account will be credited with the total number of emails for the plan. Say you upgrade to the $14.95 plan from the $9.95 plan. 1200 emails will be added to you available emails. The only math that you have to do is figure out how many more emails you need and then select a suitable plan to upgrade to. [Remember, you can always downgrade your plan anytime to go back to your regular volume for the next month.]

Say more than before on the subject of your email so that you can say it well!

Blastwizard now accepts a longer subject line than before. Now that you have more room do remember that some email clients truncate the subject line. Always test your emails in different clients before you activate the job.

Uniform 24 hour time format during job creation and job edit

We are seeing an increasing number of international clients using BlastWizard. To keep it easy for everyone to deal with timezones, we have standardized the time format to be 24 hour format when you are scheduling the job /editing the job. Please remember that all times fields are in Pacific Time. Internationalization of BlastWizard is on our roadmap and we will keep you posted on that.

As always the team at Boomerang is tuned in for your comments and feedback. Please feel free to write to us at onlinesupport@boomerang.com