Frequently Asked Questions

General

DNS

SpamAssassin

General

How do I request removal of a site from a SURBL list?

To request removal from a SURBL list, please start with the the SURBL Lookup page and follow the instructions on the removal form.

How can I keep up with important changes or updates to SURBL intelligence?

To keep up with important updates, please subscribe to the SURBL Announcement list. This is a low-volume, announcement-only list. It is important that you receive announcements such as planned changes to the rsync directory structure, zone data file layout, etc.

What testpoints are in the SURBL data?

Previously, each reputation zone file, including the bitmask-combined list multi.surbl.org, had testpoints with A records resolving to 127.0.0.2. As of about November 2005 multi has testpoints corresponding to all ones set for each bitmask position. So testpoints for multi.surbl.org now have an A record value of 127.0.0.126 instead of 127.0.0.2 . (If a new list is added in future using the 128th bit, then the new testpoint values in multi will be 127.0.0.254 . Following tradition, the first bit is not used.)

This may work better with applications that decode the bits into individual list results, but it is a change from before and it may break other applications' use of the testpoints.

The multi.surbl.org BIND zone file contains:

test.surbl.org	604800	IN	A	127.0.0.126
	604800	IN	TXT	"multi.surbl.org permanent test point"
test.multi.surbl.org	604800	IN	A	127.0.0.126
	604800	IN	TXT	"multi.surbl.org permanent test point"
surbl-org-permanent-test-point.com	604800	IN	A	127.0.0.126
	604800	IN	TXT	"multi.surbl.org permanent test point"
2.0.0.127	604800	IN	A	127.0.0.126
	604800	IN	TXT	"multi.surbl.org permanent test point"

The multi.surbl.org rbldnsd zone file contains:

test.surbl.org	:126:multi.surbl.org permanent test point
test.multi.surbl.org	:126:multi.surbl.org permanent test point
surbl-org-permanent-test-point.com	:126:multi.surbl.org permanent test point
2.0.0.127	:126:multi.surbl.org permanent test point

Those resolve into the following Address records:

Name:     test.surbl.org.multi.surbl.org
  Address:  127.0.0.126

  Name:     test.multi.surbl.org.multi.surbl.org
  Address:  127.0.0.126

  Name:     surbl-org-permanent-test-point.com.multi.surbl.org
  Address:  127.0.0.126

  Name:     2.0.0.127.multi.surbl.org
  Address:  127.0.0.126

But note that only the last, two-level domain surbl-org-permanent-test-point.com will work as the base domain for a URI in a test message for SpamAssassin. This is because URIs with test.multi.surbl.org.multi.surbl.org, etc., won't be detected by most programs because they're supposed to be reduced down to a two-level domain which would be surbl.org for those.

What URIs should a SURBL test message have?

Test URLs are:

http://surbl-org-permanent-test-point.com/

or:

http://127.0.0.2/

So if you send yourself a message with any of those testpoints as URIs, the messages should match any of our  intelligence sets you are using.

How can I send or receive messages mentioning blacklisted sites?

If blacklisted URIs must be mentioned in a message body, then one answer is to munge the URI until it's no longer parsable as a URI. E.g.,

http://somedomain.com/

can be rewritten as:

http://somedomain-MUNGED.com/

That would require some awareness on the part of the person forwarding or discussing a listed site, but it's just as doable and humanly readable as munged email addresses, which people do all the time.

Another commonly used technique is to change the "http" to something that doesn't work such as "hxxp".

It's a good practice to use little or no filtering on your security mailing list messages and abuse contact addresses, or to bypass them around filtering.

How can I report sites for possible inclusion or removal from SURBL intelligence?

Different lists use different data sources. For more information about reporting please see the Lists page.

How are redirection sites handled?

Redirection sites like drs.yahoo.com, tinyurl.com, etc. take external URIs (unfortunately including those mentioned in unsolicited messages) and redirect a browser to them. Therefore redirectors can be abused if a simple web site check only looks at the initial web site, making the whole web site appear legitimate. For example the Yahoo redirection below might be (incorrectly) parsed as a legitimate yahoo domain:

http://drs.yahoo.com/covey/parr/*http://other.address/

The big picture solution is for the redirection sites to block abusive sites on their own. In other words, they should not let abusers redirect through their sites. Some redirection sites, such as tinyurl.com, reportedly actively block and report abusers of their site. Others such as Metamark and SnipURL are using SURBL intelligence to deny abusers access to their redirection services. Here is an Open Letter to Redirection Sites that may be used or modified to contact them. And additionally we have adfditional solutions to cover with shortner abuse.

How are randomized subdomains or host names handled?

Our whitelist, which we use only to exclude domains from SURBL intelligence, not to "allow" messages, is growing but doesn't hit data from the various SURBL intelligence too often. The goal is to keep whitehats off the lists in the first place by being careful with the input data. The whitelists are intended to be a safety backstop to make sure domains with legitimate uses don't get added.

com.br, co.uk, etc., are in the SURBL whitelist. Does that mean subdomains under those won't get listed?

No. We list country code TLDs (ccTLDs) like co.uk in our whitelist as a kludge to prevent those specific types of country code two-level domains from ever getting listed. In other words, since they are on whitelists, "co.uk" and "com.br" by themselves will never appear on our lists. But somedomain.co.uk and anotherdomain.com.br will still get added to the lists just fine.

If an unsolicited message contains a site on SURBL's whitelist, does that mean it won't be detected?

No. Our whitelist is used internally to prevent certain domains and IP addresses from getting onto the lists. In that sense our whitelists are more like internal "ignore" or exclusion lists. The whitelist entries have no direct effect on the checking of messages. If a message does include some whitelisted domains, that essentially has no effect on detection using SURBL intelligence.

How can I locally whitelist some domains or IPs to prevent SURBL checking of them?

If you are using SpamAssassin or SpamCopURI, they have built-in support for local whitelists which will prevent those message body URI hosts from being checked:

# Top 125 domains whitelisted by SURBL
uridnsbl_skip_domain yahoo.com w3.org msn.com com.com yimg.com
uridnsbl_skip_domain hotmail.com doubleclick.net flowgo.com ebaystatic.com aol.com
[...]

SpamCopURI also has a built-in whitelist function:

whitelist_spamcop_uri   *.yahoo.com

Other SURBL applications may have similar exclusion features. If not, their authors may want to consider adding local whitelisting.

  1. These local whitelist entries should only apply to domains and IPs (hosts) that appear in message body URIs. They should not apply to mail server names, sender domains, message headers, etc. Therefore there's no need to add your mail server names, etc. to these whitelists. Remember that SURBL intelligence datasets are meant to opreate on message body URIs.

     

  2. To request removal from a SURBL list, please start with the the SURBL Lookup page and follow the instructions on the removal form.

Is there still code to write for the SURBL movement?

The SpamAssassin 3 plugin URIDNSBL now has generic support for domain based listings when using its urirhsbl and urirhssub commands (see the SpamAssassin section of the FAQ), so that release of SA is covered.

Devin Carraway has written a plugin for the Perl-based MTA qpsmtpd to compare domains from message body URIs against SURBL intelligence. Here's his announcement on perl.qpsmtpd, and a link to his uribl plugin. Devin's was the first MTA implementation.

Exim, sendmail, postfix, qmail, qmail-ldap and Exchange programs and plugins have been written to support our intelligence in those MTA. (Please see the Links and News pages for more information.)

DNS

I run a high volume mail server. How should I use SURBL intelligence?

For systems processing more than 250,000 messages per day, please set up a local caching name server for any of the lists you are using, including SURBL intelligence. This is considered a standard, good practice since it offloads the public name servers and improves your performance with local lookups.

A very popular and fast name server specifically meant for serving up lists is rbldnsd. SURBL zone files are available in rbldnsd format. (They are also available in BIND format, though rbldnsd is recommended as significantly leaner and faster.) There are links and instructions for using rbldnsd with rsync in the Links section.

Then arrange with the lists to get rsync access to their zone files. Since rsync only transmits differences, the zone files are kept updated in a very efficient manner. To get rsync access to SURBL zone files, please fill out our data feed access form. Other lists may have similar procedures for gaining rsync access.

Then configure your mail or SpamAssassin servers using lists to do the lookups on your local list name server. Many people run the local DNS for their lists on their mail server(s), which tends to work well since it keeps everything on the same box. If your mail server is separate from your list name server, then set up DNS on the mail server to resolve using that list name server. The following documents may be helpful in setting up local caching:

I'm using an anti-spam or anti-phishing DNS proxy, and I'm seeing legitimate sites marked as unsolicited.

There are some DNS proxy or modification services that change the responses from certain DNS queries in order to prevent users from visiting sites advertised in phishing, unsolicited messages, etc. This can cause errors when using SURBL intelligence if the proxies return an IP address of an alternative (safe) web site. The modified IP address can have an incorrect effect on SURBL list identification depending on where the bit patterns happen to be in the modified response. The result is that legitimate sites may be misidentified, but the effect appears to be somewhat random or arbitrary.

A solution is to disable such site correction or modification features on servers or clients doing SURBL queries. Alternatively, consider using regular (non-modifying) nameservers for those systems. Often the best solution is to set up a local caching nameserver.

I'm using my provider's nameservers, and I'm seeing legitimate sites marked as unsolicited.

Some ISPs such as Verizon and Charter are reportedly modifying some DNS NXDOMAIN responses in a way that causes what look like false positives on domains that are not blacklisted. Unfortunately this breaks DNS responses for SURBL intelligence and other blacklists. Please check with your ISP if you are seeing DNS responses modified in this way. Verizon has an opt-out procedure with instructions on switching to DNS servers that do not change NXDOMAIN responses. Others such as Charter have opt-out nameservers that reportedly do not support NXDOMAIN, in which case none of their nameservers may be compatible with DNS blacklists. One solution is to not use your provider's nameservers, for example by setting up your own local caching nameserver instead. Most operating systems have built-in support for running your own nameservers, and a local nameserver can significantly improve performance.

Is there a performance advantage from using name to name comparisons instead of name to IP address checks?

It's definitely quicker to do DNS lookups of the single, cached SURBL query than DNS lookups for Address and/or Name Server records for every web site in every incoming message. By comparing names to names in incoming URIs and SURBL intelligence, we avoid a major performance bottleneck of URI checks that try to resolve wild domain names, whose name servers you have no control over, into IP addresses or NS records. In other words using a SURBL is much quicker.

SpamAssassin

How do I use SURBL intelligence with SpamAssassin?

SpamAssassin 3 and later versions suport SURBL intelligence by default. See the next section for more information about enabing network tests which include SURBL checks. If you're using a version of SpamAssassin older than 3.0, then you should upgrade to the latest version to get some security patches and better performance.

For SpamAssassin 3.X, there is a suite of programs contained in the plugin URIDNSBL including some that can be used with SURBL intelligence and some that can't:

  • urirhsbl is a program that checks message body URIs against individual SURBL intelligence, i.e., it compares (mostly) domain names found in message bodies against (mostly) domain names in one SURBL per rule.
  • urirhssub is a program that checks message body URIs against the combined dataset multi  (multi.surbl.org), i.e., it compares (mostly) domain names found in message bodies against (mostly) domain names in multiple SURBL intelligence datasets.
  • uridnsbl is the SpamAssassin 3.X program that checks message body URIs against numeric lists, by resolving the URI domain names' NS records, then checking those name server IP addresses against the list. It cannot be used with SURBL intelligence, but can be used with lists of blacklisted name server IPs such as sbl.spamhaus.org.

For more information, please see SpamAssassin documentation and community.

My SpamAssassin now has SURBL support, but I'm not seeing SURBL rules hit. (Enabing network tests)

If you recently added SURBL support to SpamAssassin but are not seeing any positive hits (or other list hits), check that you have enabled network tests.

For example, if you're using spamd, make sure it's started without the -L or --local flags, which force local tests only.

If you are running Amavis, make sure amavisd.conf has $sa_local_tests_only = 0. (Uncomment this line if it was commented out before, then set the value to zero to enable network tests.)

If you are using MIMEDefang, make sure you set $SALocalTestsOnly to zero:

# If boolean true, skip SA network tests
     $SALocalTestsOnly = 0;

to enable SpamAssassin network tests from your mimedefang-filter.

Also make sure that you have a recent Net::DNS installed. Too old versions of Net::DNS seems to be a common reason for RBL checks not working, especially when upgrading from an older version of SpamAssassin.

After upgrading from SpamAssassin 3.0.0 to 3.0.1 or later, non-default URIDNSBL checks including SURBL no longer work

When upgrading from SpamAssassin 3.0.0 to 3.0.1 or later, please change rules using urirhsbl, urirhssub or uridnsbl, from rule type header to rule type body. For example:

	urirhssub URIBL_ABUSE_SURBL multi.surbl.org.   A   64
body      URIBL_ABUSE_SURBL eval:check_uridnsbl('URIBL_ABUSE_SURBL')
describe  URIBL_ABUSE_SURBL Contains a URL listed in the ABUSE SURBL list
tflags    URIBL_ABUSE_SURBL net
score     URIBL_ABUSE_SURBL 3.0

Where body above was previously header. Here is the changelog reference:

r54022 | felicity | 2004-10-07 22:21:30 +0000 (Thu, 07 Oct 2004) | 1 line

bug 3734: uridnsbl rules work on body data, not header data, so change
the rule type from header to body

faq.html version 4.01 on 05/10/2024

Data Feed Request

Our Data Feeds offer higher performance for professional users through faster updates and resulting fresher data. Freshness matters since the threat behavior is often highly dynamic, so Data Feed users can expect higher detection rates and lower false negatives.

The main data set is available in different formats:

  • API (Restfull API)
  • CSV (Comma Separated Values)
  • DNS (Private Query Service - PQS)
  • RPZ (DNS Response Policy Zones)
  • RSYNC (Sponsored Data Service)
  • RTF (JSON Real time feeds)

Rsync and DNS are typically used for mail filtering and RPZ for web filtering. High-volume systems and non-filter uses such as security research should use rsync.

For more information, please contact your reseller or see the references in Links.

Sign up for Data Feed Access.

  • Sign up for data feed access

    Direct data feed access offers better filtering performance with fresher data than is available on the public mirrors. Sign up for Data Feed Access.

  • Supported Applications

    • Apache SpamAssassin - #1 Open-Source Spam Filter
    • milter-link - filter for Sendmail and Postfix
    • More...

  • Learn more ...

    • AB - Abuse and spam sites
    • PH - Phishing sites
    • MW - Malware sites
    • More...