Better whitelisting with DNSWL

[Henrik K]

Hi,

I'd like to encourage people to take more advantage of DNSWL.

I'm currently converting DNSWL entries into trusted_networks and using
shortcircuited ALL_TRUSTED to reduce unnecessary processing. Also DNS checks
are reduced.

With only 'med' and 'high' entries, 15% of my traffic hits ALL_TRUSTED.
I just also added 'low' to the mix, since it seems to have pretty much
identical ratio with 'med'. It doubled the entry-count, so have to wait and
see how much it helps. In future the list will be just getting bigger..

This should be quite safe, since the spam occurences that I see from
whitelisted relays come mostly through some mailing-list things (poster IP
should be in Received-headers, thus ALL_TRUSTED wont hit). Also there is
some tiny amount of backscatter, but I use amavisd-new bounce killer feature
outside of SA for that.

Of course one could skip SA completely when receiving from a whitelisted
relay. It would perhaps pass little more spam, but it would be the optimal
setup performance wise.

Getting this to work in SA requires some customizing due to some
deficiencies. Don't try if you are not comfortable patching. You can use
3.2 and 3.3 (SVN). Memory usage is raised about 10-20MB.

1)

Make sure your internal_networks is set up, so external checks (relay
connecting to your MX and beyond) are done correctly. trusted_networks also
needs to contain all the same internal_networks entries. This is basic
stuff, check wiki etc.

2)

To prevent helo and dynamic checks getting borked, you need to always run
this after using sa-update:

perl -i -pe 's/Relays-Untrusted/Relays-External/' /var/spamassassin/*/*org/*

Some rules check untrusted when they should be checking external. Waiting to
be fixed, https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5856

3)

So trusted relays are never checked in DNSBLs (like they are not supposed
to), apply "DNSEval.pm fixes" patch for 3.2 or the SVN ones for 3.3 from
https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5856

4)

Comment out the offending line from NetSet.pm add_cidr(). Checking for
duplicate entries is too intensive for big lists:

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5931

(Note that it probably breaks exclusions "!1.2.3.4" if you are using such)

5)

To optionally reduce unneeded DNS queries on shortcircuiting, you can use
this hack:

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5930

6)

cf:

priority ALL_TRUSTED -10000
score ALL_TRUSTED -10
shortcircuit ALL_TRUSTED on

7)

Rsync dnswl data (see their page for info) and convert it:

perl -ne 'print "trusted_networks $1\n" if /^(\d[^;]+);\d+;(?:low|med|high);/i' \
 < generic-dnswl > dnswl_trusted.cf



That's it.. if you are trying, maybe give a shout here how it works.

It would be nice to see something like this built into SA in the future,
possibly even distributing all the entries daily with sa-update. In the
grand scale, I guess it would be easier to "enforce" users into this than
expecting them to do it at MTA level. Resources and FPs saved for everyone.

Cheers,
Henrik

[Matus UHLAR - fantomas]

On 03.07.08 11:35, Henrik K wrote:
> I'd like to encourage people to take more advantage of DNSWL.

while DNSWL('s) may be good, I encountered many cases whan spam and bounces
won't get catched by SA because the sender is in DNSQL.

> I'm currently converting DNSWL entries into trusted_networks and using
> shortcircuited ALL_TRUSTED to reduce unnecessary processing. Also DNS
> checks are reduced.

Couldn't that lead to misfiring of some RBL checks?
--
Matus UHLAR - fantomas, uhlar@... ; http://www.fantomas.sk/
Warning: I wish NOT to receive e-mail advertising to this address.
Varovanie: na tuto adresu chcem NEDOSTAVAT akukolvek reklamnu postu.
I feel like I'm diagonally parked in a parallel universe.

[Henrik K]

On Thu, Jul 03, 2008 at 10:48:07AM +0200, Matus UHLAR - fantomas wrote:
> On 03.07.08 11:35, Henrik K wrote:
> > I'd like to encourage people to take more advantage of DNSWL.
>
> while DNSWL('s) may be good, I encountered many cases whan spam and bounces
> won't get catched by SA because the sender is in DNSQL.

Be more specific. What is "many cases" in percentage of your traffic? There
will always be FPs, but if the amount is insignificant it doesn't matter.
You can try some workaround rules for clear cases.

> > I'm currently converting DNSWL entries into trusted_networks and using
> > shortcircuited ALL_TRUSTED to reduce unnecessary processing. Also DNS
> > checks are reduced.
>
> Couldn't that lead to misfiring of some RBL checks?

Not when the rules check the border (external/trusted) that they are
supposed to.

[Richard Johnson-5]

Hello, this is the first time I've used a 'list' so if I have posted in the wrong context or using the wrong protocol please forgive my blunder.
Can someone help me with Spamassassin?

Here is an overview of where I am at:

I'm running Ubuntu gutsy and have spamassassin installed. The original plan was to use it with Evolution. However, when I selected it as a plugin within Evolution it put every item of mail into the 'junk' folder and there were no spamassassin headers. My thinking was, let me test spamassassin is working at all before I get into MTA's, MUA's etc.

So, I ran this:
spamassassin --lint

and ended up with:

[9086] dbg: bayes: untie-ing
check: no loaded plugin implements 'check_main': cannot scan! at /usr/share/perl5/Mail/SpamAssassin/PerMsgStatus.pm line 164.

I then ran:
spamassassin -D --lint

and got:

[9086] dbg: logger: adding facilities: all
[9086] dbg: logger: logging level is DBG
[9086] dbg: generic: SpamAssassin version 3.2.4
[9086] dbg: config: score set 0 chosen.
[9086] dbg: util: running in taint mode? yes
[9086] dbg: util: taint mode: deleting unsafe environment variables, resetting PATH
[9086] dbg: util: PATH included '/usr/local/sbin', keeping
[9086] dbg: util: PATH included '/usr/local/bin', keeping
[9086] dbg: util: PATH included '/usr/sbin', keeping
[9086] dbg: util: PATH included '/usr/bin', keeping
[9086] dbg: util: PATH included '/sbin', keeping
[9086] dbg: util: PATH included '/bin', keeping
[9086] dbg: util: PATH included '/usr/games', keeping
[9086] dbg: util: final PATH set to: /usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/usr/games
[9086] dbg: dns: no ipv6
[9086] dbg: dns: is Net::DNS::Resolver available? yes
[9086] dbg: dns: Net::DNS version: 0.60
[9086] dbg: diag: perl platform: 5.008008 linux
[9086] dbg: diag: module installed: Digest::SHA1, version 2.11
[9086] dbg: diag: module installed: HTML::Parser, version 3.56
[9086] dbg: diag: module installed: Net::DNS, version 0.60
[9086] dbg: diag: module installed: MIME::Base64, version 3.07
[9086] dbg: diag: module installed: DB_File, version 1.814
[9086] dbg: diag: module installed: Net::SMTP, version 2.29
[9086] dbg: diag: module installed: Mail::SPF, version v2.005
[9086] dbg: diag: module not installed: Mail::SPF::Query ('require' failed)
[9086] dbg: diag: module not installed: IP::Country::Fast ('require' failed)
[9086] dbg: diag: module not installed: Razor2::Client::Agent ('require' failed)
[9086] dbg: diag: module not installed: Net::Ident ('require' failed)
[9086] dbg: diag: module not installed: IO::Socket::INET6 ('require' failed)
[9086] dbg: diag: module not installed: IO::Socket::SSL ('require' failed)
[9086] dbg: diag: module installed: Compress::Zlib, version 2.005
[9086] dbg: diag: module installed: Time::HiRes, version 1.86
[9086] dbg: diag: module not installed: Mail::DomainKeys ('require' failed)
[9086] dbg: diag: module not installed: Mail::DKIM ('require' failed)
[9086] dbg: diag: module installed: DBI, version 1.57
[9086] dbg: diag: module installed: Getopt::Long, version 2.35
[9086] dbg: diag: module installed: LWP::UserAgent, version 2.033
[9086] dbg: diag: module installed: HTTP::Date, version 1.47
[9086] dbg: diag: module installed: Archive::Tar, version 1.31
[9086] dbg: diag: module installed: IO::Zlib, version 1.04
[9086] dbg: diag: module not installed: Encode::Detect ('require' failed)
[9086] dbg: ignore: using a test message to lint rules
[9086] dbg: config: using "/etc/spamassassin" for site rules pre files
[9086] dbg: config: using "/usr/share/spamassassin" for sys rules pre files
[9086] dbg: config: using "/usr/share/spamassassin" for default rules dir
[9086] dbg: config: read file /usr/share/spamassassin/10_default_prefs.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_advance_fee.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_body_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_compensate.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_dnsbl_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_drugs.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_dynrdns.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_fake_helo_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_head_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_html_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_imageinfo.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_meta_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_net_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_phrases.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_porn.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_ratware.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_uri_tests.cf
[9086] dbg: config: read file /usr/share/spamassassin/20_vbounce.cf
[9086] dbg: config: read file /usr/share/spamassassin/23_bayes.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_accessdb.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_antivirus.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_asn.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_dcc.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_dkim.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_domainkeys.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_hashcash.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_pyzor.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_razor2.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_replace.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_spf.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_textcat.cf
[9086] dbg: config: read file /usr/share/spamassassin/25_uribl.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_de.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_fr.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_it.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_nl.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_pl.cf
[9086] dbg: config: read file /usr/share/spamassassin/30_text_pt_br.cf
[9086] dbg: config: read file /usr/share/spamassassin/50_scores.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_awl.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_shortcircuit.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_whitelist.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_whitelist_dk.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_whitelist_dkim.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_whitelist_spf.cf
[9086] dbg: config: read file /usr/share/spamassassin/60_whitelist_subject.cf
[9086] dbg: config: read file /usr/share/spamassassin/72_active.cf
[9086] dbg: config: using "/etc/spamassassin" for site rules dir
[9086] dbg: config: using "/home/user1/.spamassassin/user_prefs" for user prefs file
[9086] dbg: config: read file /home/user1/.spamassassin/user_prefs
[9086] dbg: rules: __MO_OL_9B90B merged duplicates: __MO_OL_C65FA
[9086] dbg: rules: __XM_OL_22B61 merged duplicates: __XM_OL_A842E
[9086] dbg: rules: __MO_OL_07794 merged duplicates: __MO_OL_8627E __MO_OL_F3B05
[9086] dbg: rules: __XM_OL_07794 merged duplicates: __XM_OL_25340 __XM_OL_3857F __XM_OL_4F240 __XM_OL_58CB5 __XM_OL_6554A __XM_OL_812FF __XM_OL_C65FA __XM_OL_CF0C0 __XM_OL_F475E __XM_OL_F6D01
[9086] dbg: rules: FH_MSGID_01C67 merged duplicates: __MSGID_VGA
[9086] dbg: rules: FS_NEW_SOFT_UPLOAD merged duplicates: HS_SUBJ_NEW_SOFTWARE
[9086] dbg: rules: __FH_HAS_XMSMAIL merged duplicates: __HAS_MSMAIL_PRI
[9086] dbg: rules: __MO_OL_015D5 merged duplicates: __MO_OL_6554A
[9086] dbg: rules: __MO_OL_91287 merged duplicates: __MO_OL_B30D1 __MO_OL_CF0C0
[9086] dbg: rules: KAM_STOCKOTC merged duplicates: KAM_STOCKTIP15 KAM_STOCKTIP20 KAM_STOCKTIP21 KAM_STOCKTIP4 KAM_STOCKTIP6
[9086] dbg: rules: __XM_OL_015D5 merged duplicates: __XM_OL_4BF4C __XM_OL_4EEDB __XM_OL_5B79A __XM_OL_9B90B __XM_OL_ADFF7 __XM_OL_B30D1 __XM_OL_B4B40 __XM_OL_BC7E6 __XM_OL_F3B05 __XM_OL_FF5C8
[9086] dbg: rules: __MO_OL_22B61 merged duplicates: __MO_OL_4F240 __MO_OL_ADFF7
[9086] dbg: rules: __MO_OL_812FF merged duplicates: __MO_OL_BC7E6
[9086] dbg: rules: __MO_OL_25340 merged duplicates: __MO_OL_4EEDB __MO_OL_7533E
[9086] dbg: rules: __MO_OL_58CB5 merged duplicates: __MO_OL_B4B40
[9086] dbg: rules: __DOS_HAS_ANY_URI merged duplicates: __HAS_ANY_URI
[9086] dbg: rules: __XM_OL_C9068 merged duplicates: __XM_OL_EF20B
[9086] dbg: rules: __MO_OL_72641 merged duplicates: __MO_OL_A842E
[9086] dbg: rules: __MO_OL_F475E merged duplicates: __MO_OL_FF5C8
[9086] dbg: rules: __MO_OL_4BF4C merged duplicates: __MO_OL_F6D01
[9086] dbg: conf: finish parsing
[9086] dbg: bayes: tie-ing to DB file R/O /home/user1/.spamassassin/bayes_toks
[9086] dbg: bayes: tie-ing to DB file R/O /home/user1/.spamassassin/bayes_seen
[9086] dbg: bayes: found bayes db version 3
[9086] dbg: bayes: DB journal sync: last sync: 1214824766
[9086] dbg: bayes: not available for scanning, only 1 spam(s) in bayes DB < 200
[9086] dbg: bayes: untie-ing
[9086] dbg: config: score set 0 chosen.
[9086] dbg: message: main message type: text/plain
[9086] dbg: message: ---- MIME PARSER START ----
[9086] dbg: message: parsing normal part
[9086] dbg: message: ---- MIME PARSER END ----
[9086] dbg: bayes: tie-ing to DB file R/O /home/user1/.spamassassin/bayes_toks
[9086] dbg: bayes: tie-ing to DB file R/O /home/user1/.spamassassin/bayes_seen
[9086] dbg: bayes: found bayes db version 3
[9086] dbg: bayes: DB journal sync: last sync: 1214824766
[9086] dbg: bayes: not available for scanning, only 1 spam(s) in bayes DB < 200
[9086] dbg: bayes: untie-ing
check: no loaded plugin implements 'check_main': cannot scan! at /usr/share/perl5/Mail/SpamAssassin/PerMsgStatus.pm line 164.

I've spent a number of days googling to resolve this, but I'm getting nowhere. Any pointers would be very welcome.

[Ramprasad-5]

On Thu, 2008-07-03 at 10:48 +0200, Matus UHLAR - fantomas wrote:
> On 03.07.08 11:35, Henrik K wrote:
> > I'd like to encourage people to take more advantage of DNSWL.
>
> while DNSWL('s) may be good, I encountered many cases whan spam and bounces
> won't get catched by SA because the sender is in DNSQL.

Yes , but you report that to them and they usually take care

[Matthias Leisi]

> [snip code + explanation]

Very nice :)

> It would be nice to see something like this built into SA in the future,
> possibly even distributing all the entries daily with sa-update.

We can produce almost any export format of dnswl.org data, also in a way
that it would fit for some sa-update channel. I've never looked into
sa-update channels in detail, but I'd be glad to set it up if I get some
guidance.

-- Matthias, for dnswl.org

[Marc Perkel]

Henrik K wrote:

> On Thu, Jul 03, 2008 at 10:48:07AM +0200, Matus UHLAR - fantomas wrote:
>  
>> On 03.07.08 11:35, Henrik K wrote:
>>    
>>> I'd like to encourage people to take more advantage of DNSWL.
>>>      
>>> I'm currently converting DNSWL entries into trusted_networks and using
>>> shortcircuited ALL_TRUSTED to reduce unnecessary processing. Also DNS
>>> checks are reduced.
>>>      

I strongly support your efforts. Feel free to use my white lists as
well. I have a bigger list. I also have a larger variety of "colors". On
my system white means it's 100% good. Most others white means don't
blacklist. I have "yellow lists" which means it's mixed source (not
white - not black) like yahoo, hotmail, etc. I also support both IP and
host name lookups.

http://wiki.junkemailfilter.com/index.php/Spam_DNS_Lists

And - I'm looking for domains to add to my lists.

[alexw]

Richard Johnson schrieb am 03.07.2008 10:58:

> check: no loaded plugin implements 'check_main': cannot scan! at
> /usr/share/perl5/Mail/SpamAssassin/PerMsgStatus.pm line 164.

> [9086] dbg: ignore: using a test message to lint rules
> [9086] dbg: config: using "/etc/spamassassin" for site rules pre files
> [9086] dbg: config: using "/usr/share/spamassassin" for sys rules pre files
> [9086] dbg: config: using "/usr/share/spamassassin" for default rules dir
> [9086] dbg: config: read file /usr/share/spamassassin/10_default_prefs.cf

After the 2nd line, some *.pre files should be listed. Like this:

[18926] dbg: ignore: using a test message to lint rules
[18926] dbg: config: using "/etc/mail/spamassassin" for site rules pre files
[18926] dbg: config: read file /etc/mail/spamassassin/init.pre
[18926] dbg: config: read file /etc/mail/spamassassin/v310.pre
[18926] dbg: config: read file /etc/mail/spamassassin/v312.pre
[18926] dbg: config: read file /etc/mail/spamassassin/v320.pre

They are not at your system, so it seems that the *.pre files are not
installed in the /etc/spamassassin directory. At my system (Fedora 7)
they are installed to /etc/mail/spamassassin. Look for them and move
them to /etc/spamassassin or change the site rule path
(--siteconfigpath=/some/path) to wherever your *.pre files are.

Tschau
Alex

[Richard Johnson-5]

On Fri, 2008-07-04 at 12:39 +0200, Alex Woick wrote:

> Richard Johnson schrieb am 03.07.2008 10:58:
>
> > check: no loaded plugin implements 'check_main': cannot scan! at
> > /usr/share/perl5/Mail/SpamAssassin/PerMsgStatus.pm line 164.
>
> > [9086] dbg: ignore: using a test message to lint rules
> > [9086] dbg: config: using "/etc/spamassassin" for site rules pre files
> > [9086] dbg: config: using "/usr/share/spamassassin" for sys rules pre files
> > [9086] dbg: config: using "/usr/share/spamassassin" for default rules dir
> > [9086] dbg: config: read file /usr/share/spamassassin/10_default_prefs.cf
>
> After the 2nd line, some *.pre files should be listed. Like this:
>
> [18926] dbg: ignore: using a test message to lint rules
> [18926] dbg: config: using "/etc/mail/spamassassin" for site rules pre files
> [18926] dbg: config: read file /etc/mail/spamassassin/init.pre
> [18926] dbg: config: read file /etc/mail/spamassassin/v310.pre
> [18926] dbg: config: read file /etc/mail/spamassassin/v312.pre
> [18926] dbg: config: read file /etc/mail/spamassassin/v320.pre
>
> They are not at your system, so it seems that the *.pre files are not
> installed in the /etc/spamassassin directory. At my system (Fedora 7)
> they are installed to /etc/mail/spamassassin. Look for them and move
> them to /etc/spamassassin or change the site rule path
> (--siteconfigpath=/some/path) to wherever your *.pre files are.
>
> Tschau
> Alex
>

I tried:

find / -name *.pre

no results comes back. How weird is this?