IP country plugin - can we look for two countries?

[McDonald, Dan]

I have been using the IPcountry plugin for a while, giving a slight
boost to the scores of some countries that we get more spam from and do
less work with.  I have noticed, recently, that a fair number of really
obnoxious spam is relayed between two countries before being delivered.
Is there some way to grab the metadata from IPCountry to count the
number of countries that were involved in sending a mail, and set a
score based on that?

For example, the last phish I received originated in the Netherlands and
was forwarded through Hong Kong, before delivery in the US.

Any thoughts?

--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com

[mouss-2]

McDonald, Dan wrote:

> I have been using the IPcountry plugin for a while, giving a slight
> boost to the scores of some countries that we get more spam from and do
> less work with.  I have noticed, recently, that a fair number of really
> obnoxious spam is relayed between two countries before being delivered.
> Is there some way to grab the metadata from IPCountry to count the
> number of countries that were involved in sending a mail, and set a
> score based on that?
>
> For example, the last phish I received originated in the Netherlands and
> was forwarded through Hong Kong, before delivery in the US.
>
> Any thoughts?
>
>  

you mean catching the "Junkman traveller"?

you can try

header THREE_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s/
score THREE_COUNTRIES 0.01

header FOUR_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s[a-z]{2}\s/
score FOUR_COUNTRIES 0.01

and so on...

but you need to exclude mailing-lists and resent mail, so a meta (to
detect Resent-*, List-*, ...) is better.

but even then, I am not sure this is safe. worth a test...

[mouss-2]

mouss wrote:

> McDonald, Dan wrote:
>> I have been using the IPcountry plugin for a while, giving a slight
>> boost to the scores of some countries that we get more spam from and do
>> less work with.  I have noticed, recently, that a fair number of really
>> obnoxious spam is relayed between two countries before being delivered.
>> Is there some way to grab the metadata from IPCountry to count the
>> number of countries that were involved in sending a mail, and set a
>> score based on that?
>>
>> For example, the last phish I received originated in the Netherlands and
>> was forwarded through Hong Kong, before delivery in the US.
>>
>> Any thoughts?
>>
>>  
>
> you mean catching the "Junkman traveller"?
>
> you can try
>
> header THREE_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s/
> score THREE_COUNTRIES 0.01
>
> header FOUR_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s[a-z]{2}\s/
> score FOUR_COUNTRIES 0.01
>

argh. This is wrong as it will hit things like "US US FR FR"...
> and so on...
>
> but you need to exclude mailing-lists and resent mail, so a meta (to
> detect Resent-*, List-*, ...) is better.
>
> but even then, I am not sure this is safe. worth a test...
>
>

[McDonald, Dan]

On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:

> mouss wrote:
> >> Is there some way to grab the metadata from IPCountry to count the
> >> number of countries that were involved in sending a mail, and set a
> >> score based on that?
> >
> > you mean catching the "Junkman traveller"?
> >
> > you can try
> >
> > header THREE_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s/
> > score THREE_COUNTRIES 0.01
> >
> > header FOUR_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s[a-z]{2}\s/
> > score FOUR_COUNTRIES 0.01
> >
>
> argh. This is wrong as it will hit things like "US US FR FR"...
> > and so on...

Correct.  And Spamassassin rules developers generally frown on using
back-tracking, as it slows down the whole ruleset even if only used for
one rule....

Seems like I would need to write a plugin to count this then, something
like this metacode...

my %countries
foreach my $relay (split(/\s/,$vauleofx\-relay\-countries)) {
        $countries{$relay}++;
};
return #$countries
--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com

[mouss-2]

McDonald, Dan wrote:

> On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:
>  
>> mouss wrote:
>>    
>>>> Is there some way to grab the metadata from IPCountry to count the
>>>> number of countries that were involved in sending a mail, and set a
>>>> score based on that?
>>>>        
>>> you mean catching the "Junkman traveller"?
>>>
>>> you can try
>>>
>>> header THREE_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s/
>>> score THREE_COUNTRIES 0.01
>>>
>>> header FOUR_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s[a-z]{2}\s/
>>> score FOUR_COUNTRIES 0.01
>>>
>>>      
>> argh. This is wrong as it will hit things like "US US FR FR"...
>>    
>>> and so on...
>>>      
>
> Correct.  And Spamassassin rules developers generally frown on using
> back-tracking, as it slows down the whole ruleset even if only used for
> one rule....
>
> Seems like I would need to write a plugin to count this then, something
> like this metacode...
>
> my %countries
> foreach my $relay (split(/\s/,$vauleofx\-relay\-countries)) {
> $countries{$relay}++;
> };
> return #$countries
>  

it would be good to have two other meta-headers (in addition to
X-Relay-Countries):
X-Relay-Country-Count: the total number of different countries
X-Relay-Country-Cross: the numbre of country "switches"

so "FR US FR US" would yield 2 countries but 3 switches, which should be
less "normal" than "FR FR US US".

[McDonald, Dan]

On Mon, 2008-06-30 at 22:04 +0200, mouss wrote:

> McDonald, Dan wrote:
> > On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:
> >  
> >> mouss wrote:
> >>    
> >>>> Is there some way to grab the metadata from IPCountry to count the
> >>>> number of countries that were involved in sending a mail, and set a
> >>>> score based on that?
> >>>>        
> >>> you mean catching the "Junkman traveller"?
> >>>
> >>> you can try
> >>>
> >>> header THREE_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s/
> >>> score THREE_COUNTRIES 0.01
> >>>
> >>> header FOUR_COUNTRIES X-Relay-Countries =~ /\s[a-z]{2}\s[a-z]{2}\s/
> >>> score FOUR_COUNTRIES 0.01
> >>>
> >>>      
> >> argh. This is wrong as it will hit things like "US US FR FR"...
> >>    
> >>> and so on...
> >>>      
> >
> > Correct.  And Spamassassin rules developers generally frown on using
> > back-tracking, as it slows down the whole ruleset even if only used for
> > one rule....

> it would be good to have two other meta-headers (in addition to
> X-Relay-Countries):
> X-Relay-Country-Count: the total number of different countries
> X-Relay-Country-Cross: the numbre of country "switches"
>
> so "FR US FR US" would yield 2 countries but 3 switches, which should be
> less "normal" than "FR FR US US".

Ok, how about this very messy patch to RelayCountry.pm?

--- /usr/lib/perl5/vendor_perl/5.10.0/Mail/SpamAssassin/Plugin/RelayCountry.pm  2008-01-05 15:10:50.000000000 -0600
+++ RelayCountry.pm     2008-06-30 17:11:35.000000000 -0500
@@ -76,15 +76,30 @@
   my $msg = $opts->{msg};
 
   my $countries = '';
+  my $crosscountries=0;
+  my $lcc = '';
+  my %countriesvisited;
   foreach my $relay (@{$msg->{metadata}->{relays_untrusted}}) {
     my $ip = $relay->{ip};
     my $cc = $reg->inet_atocc($ip) || "XX";
+    $crosscountries +=1 if (($lcc ne '') and ($lcc ne $cc) and ($cc ne 'XX'));
+    $countriesvisited{$cc}++ if $cc ne 'XX';
     $countries .= $cc." ";
+    $lcc = $cc if $cc ne "XX";
   }
-
+  my @visited = keys %countriesvisited;
+  my $visits = $#visited;
   chop $countries;
   $msg->put_metadata("X-Relay-Countries", $countries);
   dbg("metadata: X-Relay-Countries: $countries");
+  if ($visits++ > -1) {
+         $msg->put_metadata("X-Relay-Country-Count", $visits);
+         dbg("metadata: X-Relay-Country-Count: $visits");
+  }
+  if ($crosscountries >0) {
+         $msg->put_metadata("X-Relay-Country-Cross", $crosscountries);
+         dbg("metadata: X-Relay-Country-Cross: $crosscountries");
+  }
 
   return 1;
 }

--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com

[McDonald, Dan]

On Mon, 2008-06-30 at 17:17 -0500, McDonald, Dan wrote:

> On Mon, 2008-06-30 at 22:04 +0200, mouss wrote:
> > McDonald, Dan wrote:
> > > On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:
> > >  
> > >> mouss wrote:
> > >>    
> > >>>> Is there some way to grab the metadata from IPCountry to count the
> > >>>> number of countries that were involved in sending a mail, and set a
> > >>>> score based on that?
> > >>>>        
> > >>> you mean catching the "Junkman traveller"?
> > >>>

Ok, been fiddling with this.  Here is my current rule:

header __IS_LIST exists:List-Id
describe __IS_LIST Is this a mailing list?

header __MULTI_COUNTRY exists:X-Relay-Country-Count
describe __MULTI_COUNTRY Has this message passed through two or more countries?

header __LAST_RELAY_US X-Relay-Countries=~/US\b$/
describe __LAST_RELAY_US Came from our home country

meta AE_RELAY_MANY !__IS_LIST && __MULTI_COUNTRY && !__LAST_RELAY_US
describe AE_RELAY_MANY passed through 2 foreign countries and is not a mailing list
score AE_RELAY_MANY 0.25

I also changed RelayCountry.pm to only insert the X-Relay-Country-Count
header if there were two or more countries involved, mainly to allow a
simple exists query rather than a regex...

But I was very encouraged by my first two hits:
Jul  1 08:05:03 ca amavis[1869]: (01869-04) SPAM,
<mrsserena_wong15@...> -> <luser@...>, Yes, score=22.549
tag=-99 tag2=4.5 kill=6.31 tests=[ADVANCE_FEE_2=2.049,
ADVANCE_FEE_3=1.435, ADVANCE_FEE_4=1.502, AE_RELAY_MANY=0.1,
DATE_IN_FUTURE_06_12=3.099, DEAR_SOMETHING=2.234,
FORGED_MUA_OUTLOOK=4.199, FREEMAIL_FROM=0.5, FREEMAIL_REPLYTO=2,
L_P0F_Linux=-0.1, MSOE_MID_WRONG_CASE=0.699, RELAY_NG=2,
SARE_FRAUD_X3=1.667, US_DOLLARS_3=1.165], autolearn=disabled

Jul  1 08:13:55 ca amavis[1852]: (01852-07) SPAM, <121212@...> ->
<luser@...>, Yes, score=24.912 tag=-99 tag2=4.5 kill=6.31
tests=[ADVANCE_FEE_2=2.049, ADVANCE_FEE_3=1.435, ADVANCE_FEE_4=1.502,
AE_RELAY_MANY=0.1, DEAR_SOMETHING=2.234, FORGED_MUA_OUTLOOK=4.199,
FREEMAIL_FROM=0.5, FREEMAIL_REPLYTO=2, L_P0F_Linux=-0.1,
MSOE_MID_WRONG_CASE=0.699, RAZOR2_CF_RANGE_51_100=0.5,
RAZOR2_CF_RANGE_E4_51_100=1.5, RAZOR2_CHECK=0.5, RELAY_CN=3,
SARE_FRAUD_X3=1.667, SPF_SOFTFAIL=0.654, SUBJ_ALL_CAPS=1.806,
URG_BIZ=0.667], autolearn=disabled


--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com

[mouss-2]

McDonald, Dan wrote:

> On Mon, 2008-06-30 at 17:17 -0500, McDonald, Dan wrote:
>  
>> On Mon, 2008-06-30 at 22:04 +0200, mouss wrote:
>>    
>>> McDonald, Dan wrote:
>>>      
>>>> On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:
>>>>  
>>>>        
>>>>> mouss wrote:
>>>>>    
>>>>>          
>>>>>>> Is there some way to grab the metadata from IPCountry to count the
>>>>>>> number of countries that were involved in sending a mail, and set a
>>>>>>> score based on that?
>>>>>>>        
>>>>>>>              
>>>>>> you mean catching the "Junkman traveller"?
>>>>>>
>>>>>>            
>
> Ok, been fiddling with this.  Here is my current rule:
>
> header __IS_LIST exists:List-Id
> describe __IS_LIST Is this a mailing list?
>
> header __MULTI_COUNTRY exists:X-Relay-Country-Count
> describe __MULTI_COUNTRY Has this message passed through two or more countries?
>
> header __LAST_RELAY_US X-Relay-Countries=~/US\b$/
> describe __LAST_RELAY_US Came from our home country
>
> meta AE_RELAY_MANY !__IS_LIST && __MULTI_COUNTRY && !__LAST_RELAY_US
> describe AE_RELAY_MANY passed through 2 foreign countries and is not a mailing list
> score AE_RELAY_MANY 0.25
>
> I also changed RelayCountry.pm to only insert the X-Relay-Country-Count
> header if there were two or more countries involved, mainly to allow a
> simple exists query rather than a regex...
>
> But I was very encouraged by my first two hits:
> Jul  1 08:05:03 ca amavis[1869]: (01869-04) SPAM,
> <mrsserena_wong15@...> -> <luser@...>, Yes, score=22.549
> tag=-99 tag2=4.5 kill=6.31 tests=[ADVANCE_FEE_2=2.049,
> ADVANCE_FEE_3=1.435, ADVANCE_FEE_4=1.502, AE_RELAY_MANY=0.1,
> DATE_IN_FUTURE_06_12=3.099, DEAR_SOMETHING=2.234,
> FORGED_MUA_OUTLOOK=4.199, FREEMAIL_FROM=0.5, FREEMAIL_REPLYTO=2,
> L_P0F_Linux=-0.1, MSOE_MID_WRONG_CASE=0.699, RELAY_NG=2,
> SARE_FRAUD_X3=1.667, US_DOLLARS_3=1.165], autolearn=disabled
>
> Jul  1 08:13:55 ca amavis[1852]: (01852-07) SPAM, <121212@...> ->
> <luser@...>, Yes, score=24.912 tag=-99 tag2=4.5 kill=6.31
> tests=[ADVANCE_FEE_2=2.049, ADVANCE_FEE_3=1.435, ADVANCE_FEE_4=1.502,
> AE_RELAY_MANY=0.1, DEAR_SOMETHING=2.234, FORGED_MUA_OUTLOOK=4.199,
> FREEMAIL_FROM=0.5, FREEMAIL_REPLYTO=2, L_P0F_Linux=-0.1,
> MSOE_MID_WRONG_CASE=0.699, RAZOR2_CF_RANGE_51_100=0.5,
> RAZOR2_CF_RANGE_E4_51_100=1.5, RAZOR2_CHECK=0.5, RELAY_CN=3,
> SARE_FRAUD_X3=1.667, SPF_SOFTFAIL=0.654, SUBJ_ALL_CAPS=1.806,
> URG_BIZ=0.667], autolearn=disabled
>
>
>  

Good. I'll have to test this.

(you should open a bugzilla ticket so that this gets integrated in the
next version...).

[McDonald, Dan]

On Tue, 2008-07-01 at 18:18 +0200, mouss wrote:

> McDonald, Dan wrote:
> > On Mon, 2008-06-30 at 17:17 -0500, McDonald, Dan wrote:
> >  
> >> On Mon, 2008-06-30 at 22:04 +0200, mouss wrote:
> >>    
> >>> McDonald, Dan wrote:
> >>>      
> >>>> On Sat, 2008-06-28 at 01:40 +0200, mouss wrote:
> >>>>  
> >>>>        
> >>>>> mouss wrote:

> Good. I'll have to test this.
>
> (you should open a bugzilla ticket so that this gets integrated in the
> next version...).

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5934


--
Daniel J McDonald, CCIE #2495, CISSP #78281, CNX
Austin Energy
http://www.austinenergy.com