[Bug 5941] New: parsing original SMTP Server not working properly

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941

           Summary: parsing original SMTP Server not working properly
           Product: Spamassassin
           Version: 3.2.5
          Platform: PC
        OS/Version: Linux
            Status: NEW
          Severity: normal
          Priority: P5
         Component: spamassassin
        AssignedTo: dev@...
        ReportedBy: anyaddress@...
                CC: anyaddress@...


Hello,

I think I have hitten a bug in the way SA parses out the original SMTP host.
I send and email from my mail client (to myself) through the SMTP server of
GMX. SA thinks the Mail was sent directly from my computer (i.e. my
dsl-routers IP) without using GMXs SMTP server.

SPF_FAIL, RCVD_IN_PBL RBL, RCVD_IN_XBL RBL, RCVD_IN_SORBS_DUL and RDNS_DYNAMIC
seem to think 85.55.41.198 was the SMTP server - which is wrong. 85.55.41.198
is the IP my dsl-router uses to connect to the Internet.

The way of the mail is the following:

MUA (kmail) -> GMXs SMTP Server -> GMX forwards it from anyaddress@... to
seclinet@... (I have set it like this in my account preferences at
GMX) -> fdm (which is a similar to fetchmail) fetches the mail via pop3 ->
procmail (gets fed by fdm) -> spamassassin (called from procmail as first
rule).

Someone on the sa-users list pointed out that the following header might be the
problem and that SA might not hanlde fdm.

Received: by localhost (fdm 1.5, account "gmx");
         Mon, 14 Jul 2008 01:04:12 +0200



Full headers:

------------------------------------------------------------------------------
Return-Path: <anyaddress@...>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org
X-Spam-Level: ****
X-Spam-Status: No, score=4.3 required=5.0
tests=AWL,BAYES_40,RCVD_IN_PBL,RCVD_IN_SORBS_DUL,RCVD_IN_XBL,RDNS_DYNAMIC,SPF_FAIL,TVD_SPACE_RATIO
        autolearn=no
        bayes=0.2760
        language=
        report:
        *  0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL
        *      [85.55.41.198 listed in zen.spamhaus.org]
        *  3.0 RCVD_IN_XBL RBL: Received via a relay in Spamhaus XBL
        *  0.9 RCVD_IN_SORBS_DUL RBL: SORBS: sent directly from dynamic IP
address
        *      [85.55.41.198 listed in dnsbl.sorbs.net]
        *  0.7 SPF_FAIL SPF: sender does not match SPF record (fail)
        *      [SPF failed: Please see
http://www.openspf.org/Why?s=mfrom&id=anyaddress%40gmx.net&ip=85.55.41.198&r=momo.seclinet.org]
        * -0.2 BAYES_40 BODY: Bayesian spam probability is 20 to 40%
        *      [score: 0.2760]
        *  2.2 TVD_SPACE_RATIO BODY: TVD_SPACE_RATIO
        *  0.1 RDNS_DYNAMIC Delivered to trusted network by host with
        *      dynamic-looking rDNS
        * -3.4 AWL AWL: From: address is in the auto white-list
X-Flags: 0000
Delivered-To: GMX delivery to seclinet@...
Received: by localhost (fdm 1.5, account "gmx");
        Mon, 14 Jul 2008 01:04:12 +0200
Received: (qmail 6881 invoked by alias); 13 Jul 2008 22:29:06 -0000
Delivered-To: GMX delivery to anyaddress@...
Received: (qmail invoked by alias); 13 Jul 2008 22:29:06 -0000
Received: from 198.pool85-55-41.dynamic.orange.es (EHLO [192.168.0.25])
[85.55.41.198]
  by mail.gmx.net (mp004) with SMTP; 14 Jul 2008 00:29:06 +0200
X-Authenticated: #8384405
X-Provags-ID: V01U2FsdGVkX1/KEJsVuZLKMG4BVaXLiJgyzPl76GsqwvYJeDn+q7
        XuSbVqmMorwDIp
From: Tom Fernandes <anyaddress@...>
To: Tom Fernandes <anyaddress@...>
Subject: test-procmail
Date: Mon, 14 Jul 2008 00:29:04 +0200
User-Agent: KMail/1.9.9
MIME-Version: 1.0
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200807140029.04272.anyaddress@...>
X-FuHaFi: 0.00
X-GMX-Antivirus: 0 (no virus found)
X-GMX-Antispam: -2 (not scanned, spam filter disabled)
X-Resent-By: Forwarder <forwarder@...>
X-Resent-For: anyaddress@...
X-Resent-To: seclinet@...
X-GMX-UID: /PQbLLcNa0AodebBJTAzUog3Njh6dE7a
X-Length: 2321
X-UID: 1521
------------------------------------------------------------------------------


thanks,


Tom


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941


Justin Mason <jm@...> changed:

           What    |Removed                     |Added
----------------------------------------------------------------------------
             Status|NEW                         |RESOLVED
         Resolution|                            |WORKSFORME




--- Comment #1 from Justin Mason <jm@...>  2008-07-15 01:21:44 PST ---

> Received: by localhost (fdm 1.5, account "gmx");
>         Mon, 14 Jul 2008 01:04:12 +0200
> Received: (qmail 6881 invoked by alias); 13 Jul 2008 22:29:06 -0000
> Delivered-To: GMX delivery to anyaddress@...
> Received: (qmail invoked by alias); 13 Jul 2008 22:29:06 -0000
> Received: from 198.pool85-55-41.dynamic.orange.es (EHLO [192.168.0.25])
> [85.55.41.198]
>   by mail.gmx.net (mp004) with SMTP; 14 Jul 2008 00:29:06 +0200

The problem here is that there's no noticeable transfer from mail.gmx.net to
localhost; there's no Received line containing an IP address etc.  So it
results in a single hop, like so:

[26366] dbg: metadata: X-Spam-Relays-External: [ ip=85.55.41.198
rdns=198.pool85-55-41.dynamic.orange.es helo=!192.168.0.25! by=mail.gmx.net
ident= envfrom= intl=0 id= auth= msa=0 ]

for the dynamic-IP-to-GMX hop.

I think the best fix for you is to simply trust the GMX mailhost, and the
dynamic IP range that you are in; add them to internal_networks.  That way,
mail from your machine to your machine via GMX will always wind up trusted,
which is what you're after.

Alternatively deliver the mail locally instead of hopping via GMX (simply
install an MTA like postfix).  Or alternatively still, fix fdm to add a decent
Received line ;)


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #2 from Tom Fernandes <anyaddress@...>  2008-07-15 12:01:33 PST ---
(In reply to comment #1)
[...]

> Alternatively deliver the mail locally instead of hopping via GMX (simply
> install an MTA like postfix).  Or alternatively still, fix fdm to add a decent
> Received line ;)
>

From looking at

http://tools.ietf.org/html/rfc2821#section-4.4

I would say that fdm - to be rfc-conform - should add the "FROM" field to the
"Received:" line - and then this problem should be solved - is that correct?

If so I would forward this info to the fdm author.


thanks,


Tom


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #3 from Justin Mason <jm@...>  2008-07-16 02:41:26 PST ---
(In reply to comment #2)

> (In reply to comment #1)
> [...]
>
> > Alternatively deliver the mail locally instead of hopping via GMX (simply
> > install an MTA like postfix).  Or alternatively still, fix fdm to add a decent
> > Received line ;)
> >
>
> From looking at
>
> http://tools.ietf.org/html/rfc2821#section-4.4
>
> I would say that fdm - to be rfc-conform - should add the "FROM" field to the
> "Received:" line - and then this problem should be solved - is that correct?

If it looked more like sendmail or Postfix' format, it would be best. ;)


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #4 from Tom Fernandes <anyaddress@...>  2008-07-16 09:49:13 PST ---
Hi,

> The problem here is that there's no noticeable transfer from mail.gmx.net to
> localhost; there's no Received line containing an IP address etc.  So it
> results in a single hop, like so:
>
> [26366] dbg: metadata: X-Spam-Relays-External: [ ip=85.55.41.198
> rdns=198.pool85-55-41.dynamic.orange.es helo=!192.168.0.25! by=mail.gmx.net
> ident= envfrom= intl=0 id= auth= msa=0 ]
>
> for the dynamic-IP-to-GMX hop.

I doubt that this is really the problem. I did 2 checks:

1) I checked another mail which has also been send from a dynamic IP through a
proper SMTP server (in this case web.de). This time SA doesn't think that the
original SMTP server is the dynamic IP address it looks like (when looking at
X-Spam-Status).
fdm's "Received:" line is in both mails is exactly the same except for the
timestamp.

2) I tested what happens if I remove the following lines (the ones added by
fdm)

---------------------------------------------------------
Received: by localhost (fdm 1.5, account "gmx");
  Sat, 12 Jul 2008 ...
---------------------------------------------------------

from the mail below and from the one in my first post and run them through SA
again.
Result ist that in the mail below the original SMTP gets found and in the other
one SA still thinks the dynamic IP of the dsl-modem is the original SMTP
server.


For testing purpose I switched the original IP 85.55.41.198 (of the senders
dsl-modem) with 81.41.72.156 (which is the one from the first post) in the
source of the mail just to be sure that the different IP won't make a
difference. I also substituted the original surname of the sender with xxx.

Full headers:

-------------------------------------------------------------------------
Return-Path: <assis.xxx@...>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org
X-Spam-Level:
X-Spam-Status: No, score=-3.6 required=5.0 tests=BAYES_00
        autolearn=ham
        bayes=0.0000
        language=de
        report:
        * -3.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
        *      [score: 0.0000]
X-Flags: 1001
Delivered-To: GMX delivery to seclinet@...
Received: by localhost (fdm 1.5, account "gmx");
  Sat, 12 Jul 2008 18:07:54 +0000
Received: (qmail 19718 invoked by alias); 5 Jul 2008 21:12:56 -0000
Delivered-To: GMX delivery to anyaddress@...
Received: (qmail invoked by alias); 05 Jul 2008 21:12:56 -0000
Received: from fmmailgate01.web.de (EHLO fmmailgate01.web.de) [217.72.192.221]
  by mx0.gmx.net (mx005) with SMTP; 05 Jul 2008 23:12:56 +0200
Received: from smtp08.web.de (fmsmtp08.dlan.cinetic.de [172.20.5.216])
  by fmmailgate01.web.de (Postfix) with ESMTP id 5C447E69BF81
  for <anyaddress@...>; Sat,  5 Jul 2008 23:12:56 +0200 (CEST)
Received: from [85.55.41.198] (helo=[85.55.41.198])
  by smtp08.web.de with asmtp (TLSv1:RC4-MD5:128)
  (WEB.DE 4.109 #226)
  id 1KFF3v-0003u5-00
  for anyaddress@...; Sat, 05 Jul 2008 23:12:56 +0200
Subject: Re: Wegbeschreibung
From: Assis <assis.xxx@...>
To: Tom Fernandes <anyaddress@...>
In-Reply-To: <200807051433.43420.anyaddress@...>
References: <200807051433.43420.anyaddress@...>
Content-Type: text/plain;
  charset=utf-8
Date: Sat, 05 Jul 2008 22:26:55 +0200
Message-Id: <1215289615.8598.13.camel@...>
Mime-Version: 1.0
X-Mailer: Evolution 2.10.3
Content-Transfer-Encoding: quoted-printable
Sender: assis.xxx@...
X-Sender: assis.xxx@...
X-GMX-Antivirus: 0 (no virus found)
X-GMX-Antispam: -2 (not scanned, spam filter disabled)
X-Resent-By: Forwarder <forwarder@...>
X-Resent-For: anyaddress@...
X-Resent-To: seclinet@...
X-GMX-UID: 4dtZAbI2QEVtHLCYOXRpsxJKNzg2NYJu
X-Length: 2506
X-UID: 1518

--------------------------------------------------------------------------


thanks,


Tom


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #5 from Justin Mason <jm@...>  2008-07-17 03:44:09 PST ---
I don't understand what you're saying.  the fdm handover is invisible to SA --
that's the problem.

Thinking about it more, I think the correct action for you to take is to trust
the GMX relays and your own address in the dialup/dynamic address pool you're
using, as I suggested.  fundamentally, even if fdm fixes their Received lines,
the issue will still be that the mail is originating from a dialup pool -- but
the key fact is that it's *your* host in the dialup pool, and that needs to be
trusted.


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #6 from Karsten Bräckelmann <guenther@...>  2008-07-17 04:54:51 PST ---
In a case like this, fixing fdm won't help. The Spamhaus hits are the very same
when using fetchmail, for example, which is perfectly identified by SA.

The issue is, that you indeed sent your mail via a single hop, aka direct MUA
to MX. This is because your outgoing SMTP actually *is* the final recipients
MX. You did send a mail from your dialup IP directly to the recipients MX. SA
correctly checks that IP against the blacklists, and Spamhaus is punishing
you...

FWIW, I'm not convinced that trusting the entire Orange dialup IP space is a
proper workaround. ;)


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #7 from Justin Mason <jm@...>  2008-07-17 05:57:34 PST ---
(In reply to comment #6)
> FWIW, I'm not convinced that trusting the entire Orange dialup IP space is a
> proper workaround. ;)

actually, here's what might be better -- if you can login to the outbound relay
using SMTP-AUTH SA should be able to detect that, and avoid applying those
rules.


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #8 from Tom Fernandes <anyaddress@...>  2008-07-17 06:34:10 PST ---
Hi,

(In reply to comment #7)
> actually, here's what might be better -- if you can login to the outbound relay
> using SMTP-AUTH SA should be able to detect that, and avoid applying those
> rules.

Test case 1) - headers below
Sent mail with MUA from dynamic IP with SMTP-AUTH through GMX's SMTP server to
my own GMX address.
Result: SA thinks the dynamic IP is the originating SMTP server

Test case 2) - headers below
Sent mail with MUA from dynamic IP with SMTP-AUTH through web.de's SMTP server
to my own web.de address.
Result: SA doesn't think the dynamic IP is the originating SMTP server


Removing the "Received:" lines added by fdm didn't make any changes.

I also went through some old mails that where send to me by people having an
GMX account and using a MUA, and found out that in all of them SA confused the
SMTP server to be dynamic IP address.



Full headers:

gmx headers ----------------------------------------------------------------
Return-Path: <seclinet@...>
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org
X-Spam-Level: *
X-Spam-Status: No, score=1.6 required=5.0
tests=AWL,BAYES_05,RCVD_IN_PBL,RCVD_IN_SORBS_DUL,RDNS_DYNAMIC,SPF_FAIL
        autolearn=no
        bayes=0.0402
        language=
        report:
        *  0.9 RCVD_IN_PBL RBL: Received via a relay in Spamhaus PBL
        *      [85.55.47.166 listed in zen.spamhaus.org]
        *  0.9 RCVD_IN_SORBS_DUL RBL: SORBS: sent directly from dynamic IP
address
        *      [85.55.47.166 listed in dnsbl.sorbs.net]
        *  0.7 SPF_FAIL SPF: sender does not match SPF record (fail)
        *      [SPF failed: Please see
http://www.openspf.org/Why?s=mfrom&id=seclinet%40gmx.net&ip=85.55.47.166&r=momo.seclinet.org]
        * -1.1 BAYES_05 BODY: Bayesian spam probability is 1 to 5%
        *      [score: 0.0402]
        *  0.1 RDNS_DYNAMIC Delivered to trusted network by host with
        *      dynamic-looking rDNS
        *  0.1 AWL AWL: From: address is in the auto white-list
X-Flags: 0000
Delivered-To: GMX delivery to seclinet@...
Received: by localhost (fdm 1.5, account "gmx");
  Thu, 17 Jul 2008 15:08:29 +0200
Received: (qmail invoked by alias); 17 Jul 2008 13:08:12 -0000
Received: from 166.pool85-55-47.dynamic.orange.es (EHLO [192.168.0.25])
[85.55.47.166]
  by mail.gmx.net (mp062) with SMTP; 17 Jul 2008 15:08:12 +0200
X-Authenticated: #22064775
X-Provags-ID: V01U2FsdGVkX19nTCR2S3ZnsGjQJEGZie3oRRwaX6FrXXjL3Di4py
  qYRO9qhe6E6KFo
From: Tom Fernandes <seclinet@...>
To: Tom Fernandes <seclinet@...>
Subject: test from gmx to gmx
Date: Thu, 17 Jul 2008 15:08:10 +0200
User-Agent: KMail/1.9.9
MIME-Version: 1.0
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200807171508.10972.seclinet@...>
X-FuHaFi: 0.00
X-GMX-Antivirus: 0 (no virus found)
X-GMX-Antispam: -2 (not scanned, spam filter disabled)
X-GMX-UID: D31McHM3MmApdezCO2BngwIxMjQ1N52M
X-Length: 1949
X-UID: 1529

web.de headers --------------------------------------------------------
X-Spam-Checker-Version: SpamAssassin 3.2.5 (2008-06-10) on momo.seclinet.org
X-Spam-Level:
X-Spam-Status: No, score=-5.4 required=5.0 tests=ALL_TRUSTED,BAYES_00
        autolearn=unavailable
        bayes=0.0000
        language=
        report:
        * -1.8 ALL_TRUSTED Passed through trusted hosts only via SMTP
        * -3.6 BAYES_00 BODY: Bayesian spam probability is 0 to 1%
        *      [score: 0.0000]
Received: by localhost (fdm 1.5, account "web");
  Fri, 18 Jul 2008 12:58:22 +2400
Received: from [85.55.47.166] (helo=[192.168.0.25])
  by smtp05.web.de with asmtp (TLSv1:AES256-SHA:256)
  (WEB.DE 4.109 #226)
  id 1KJSul-0002uZ-00
  for carteras@...; Thu, 17 Jul 2008 14:48:55 +0200
From: Tom Fernandes <carteras@...>
To: carteras@...
Subject: test from web.de to web.de
Date: Thu, 17 Jul 2008 14:48:44 +0200
User-Agent: KMail/1.9.9
MIME-Version: 1.0
Content-Type: text/plain;
  charset="utf-8"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline
Message-Id: <200807171448.44996.carteras@...>
Sender: carteras@...
X-Sender: carteras@...
X-Length: 1051
X-UID: 146

----------------------------------------------------------------------

If this is really an SA problem I will look into SA code on the weekend and see
if can come up with a patch.

thanks so far,


Tom


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.

https://issues.apache.org/SpamAssassin/show_bug.cgi?id=5941





--- Comment #9 from Tom Fernandes <anyaddress@...>  2008-08-03 08:01:38 PST ---
After reading in lib/Mail/SpamAssassin/Message/Metadata/Received.pm and doing
some tests I think that the issue is that my SMTP-AUTH with the GMX SMTP server
is not recognized by SA.

AFAICT there is nothing in GMXs received headers where SA can tell from that
I've been authenticating with GMX before sending my mail. It looks like GMX
adds the "X-Authenticated" header for that but SA can't rely on that as it can
be easily forged.

When I modify GMXs received header in the mail manually (switch SMTP with
ASMTP) and the according part in Received.pm (the line where GMXs mailserver is
matched) so that SA thinks that I've been authenticated before relaying through
GMX, the connection is trusted and the SPF and other rules are not run on the
dynamic IP.

It looks like this is a problem for all GMX users, using a pop fetcher + SA
when receiving mails from somebody sending from a GMX account using a dialup
IP. It does not make a difference if the sender uses SMTP or GMX webfrontend
for sending. In both cases SA doesn't find out that the user authenticated
before using GMX.

Wouldn't it be possible to trust GMX server if it is the first hop in the
received headers and it's not listed as an MX?. AFAICT GMX is known to
authenticate all users who are relaying through them and the MX servers don't
allow relaying (I did a fast telnet-check).

If the headers of a mail send through the webfrontend is of any help - let me
know.

thanks,


Tom


--
Configure bugmail: https://issues.apache.org/SpamAssassin/userprefs.cgi?tab=email
------- You are receiving this mail because: -------
You are the assignee for the bug.