Mailman not recognizing some bounces?

[smontanaro]

I've been working on the SpamBayes filtering on mail.python.org for a couple
weeks.  Now that the training database has matured a bit for the most part
the unsures I see now seem to be bounce messages for a handful of mailing
list subscribers which Mailman doesn't handle automatically.  The most
problematic domains are rediffmail.com, big.or.jp, 163.com, ospaz.ru and
126.com.  The bounce messages from 163.com and 126.com (and I think
big.or.jp) are written in the native character sets (Chinese or Japanese),
so I can understand if Mailman doesn't recognize them.  I'm not sure what
the problem is with the others.

Is this a known issue with Mailman?  Are there workarounds?  We're running
2.1.10.

Thanks,

--
Skip Montanaro - skip@... - http://www.webfast.com/~skip/
------
ELON MUSK: If fuel cells were good, don't think you'd see them somewhere,
like maybe in a laptop or a cell phone or a $200 million military satellite
maybe?  And yet, where do you see them?
SPENCER MICHELS: You don't.
ELON MUSK: Exactly.
------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[Mark Sapiro-3]

skip@... wrote:
>
>I've been working on the SpamBayes filtering on mail.python.org for a couple
>weeks.  Now that the training database has matured a bit for the most part
>the unsures I see now seem to be bounce messages for a handful of mailing
>list subscribers which Mailman doesn't handle automatically.


I don't understand. Are you talking about the incoming message to
listname-bounces@... or the unrecognized bounce notification
from Mailman to the list owner?

If it's the incoming message, Mailman hasn't had a chance to recognize
it yet, and any changes to Mailman's bounce recognition isn't going to
affect what your Bayes filter sees.


>The most
>problematic domains are rediffmail.com, big.or.jp, 163.com, ospaz.ru and
>126.com.  The bounce messages from 163.com and 126.com (and I think
>big.or.jp) are written in the native character sets (Chinese or Japanese),
>so I can understand if Mailman doesn't recognize them.  I'm not sure what
>the problem is with the others.
>
>Is this a known issue with Mailman?  Are there workarounds?  We're running
>2.1.10.


It is a known issue in that

1) Many MTA's do not send RFC 3464 compliant DSNs
2) We do our best to recognize non-compliant DSN's, but humans are
   better at it than is code, so there will always be 'unrecognized'
   bounces.
3) We make no attempt to recognize any language other than English.

If you send me example DSNs from the above domains, I can try to add
recognition for them. I tried mailing to
bogus_address_test_dsn@...,
bogus_address_test_dsn@..., bogus_address_test_dsn@...,
bogus_address_test_dsn@... and bogus_address_test_dsn@... to
see if I could get DSN's from these domains, but rediffmail.com,
163.com and 126.com rejected the recipient at SMTP time and the others
accepted the mail and haven't yet returned a DSN.

The best 'workaround' as far as recognizing DSNs in Mailman is
concerned is to enable Mailman's VERP options

VERP_PASSWORD_REMINDERS = Yes
VERP_PERSONALIZED_DELIVERIES = Yes
VERP_DELIVERY_INTERVAL = 1

--
Mark Sapiro <mark@...>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[Brad Knowles-3]

skip@... wrote:

> I've been working on the SpamBayes filtering on mail.python.org for a couple
> weeks.  Now that the training database has matured a bit for the most part
> the unsures I see now seem to be bounce messages for a handful of mailing
> list subscribers which Mailman doesn't handle automatically.  The most
> problematic domains are rediffmail.com, big.or.jp, 163.com, ospaz.ru and
> 126.com.  The bounce messages from 163.com and 126.com (and I think
> big.or.jp) are written in the native character sets (Chinese or Japanese),
> so I can understand if Mailman doesn't recognize them.  I'm not sure what
> the problem is with the others.

That would be a question primarily for Mark Sapiro, Tokio Kikuchi, or Barry
Warsaw.

> Is this a known issue with Mailman?  Are there workarounds?  We're running
> 2.1.10.

I know there are some bounce formats that Mailman doesn't properly handle,
because I get them as the primary active listowner for the various mailman-*
mailing lists on python.org, but I've been so busy that I haven't tried to
do much of anything with them.

I will be installing 2.1.11 soon.

--
Brad Knowles <brad@...>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[Brad Knowles-3]

Mark Sapiro wrote:

> The best 'workaround' as far as recognizing DSNs in Mailman is
> concerned is to enable Mailman's VERP options
>
> VERP_PASSWORD_REMINDERS = Yes
> VERP_PERSONALIZED_DELIVERIES = Yes
> VERP_DELIVERY_INTERVAL = 1

The last item here had been set to 10, but I have reduced it to 1 and
restarted Mailman.

Otherwise, I think we're already doing about as much as we can at the site
level, it's up to the individual listowners to choose whether or not to do
personalization/VERP.


Of course, as the primary active listmaster/postmaster/Mailman site admin, I
can help our listowners implement these features, but I generally try to
avoid stepping on any toes of any listowners unless I feel I really have no
other choice.

--
Brad Knowles <brad@...>
LinkedIn Profile: <http://tinyurl.com/y8kpxu>
------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[Mark Sapiro-3]

Mark Sapiro wrote:


>
>If you send me example DSNs from the above domains, I can try to add
>recognition for them. I tried mailing to
>bogus_address_test_dsn@...,
>bogus_address_test_dsn@..., bogus_address_test_dsn@...,
>bogus_address_test_dsn@... and bogus_address_test_dsn@... to
>see if I could get DSN's from these domains, but rediffmail.com,
>163.com and 126.com rejected the recipient at SMTP time and the others
>accepted the mail and haven't yet returned a DSN.


Skip replied to me off list with example DSNs from rediffmail.com and
163.com. Both these DSNs were recognized by Mailman 2.1.11 which has
some improvements over 2.1.10 so I'm not sure if they were recognized
in  2.1.10.

However, Skip also wrote:

>I'm referring to the delivery status notification which is sent to, say,
>python-list-bounces+xyz=rediffmail.com@....  I realize that Mailman
>has yet to see it, but I assume it eventually will see it and should disable
>the subscription after N bounces.  One rediffmail.com email address in
>particular has piled up hundreds of bounces without being disabled.


If the bounces are actually being returned to
python-list-bounces+xyz=rediffmail.com@... as opposed to
python-list-bounces@..., then they are VERPed and they should
be recognized and scored as a bounce for xyz@... regardless
of the content of the DSN.

Check Mailman's bounce log to see if they are being scored.

There is a potential issue. If you have

VERP_PROBES = Yes

in mm_cfg.py and for some reason the VERP probes aren't bouncing or are
not being recognized, the user will never have delivery disabled.

--
Mark Sapiro <mark@...>        The highway is for gamblers,
San Francisco Bay Area, California    better use your sense - B. Dylan

------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[Stephen J. Turnbull]

skip@... writes:

 > The most problematic domains are rediffmail.com, big.or.jp,
 > 163.com, ospaz.ru and 126.com.  The bounce messages from 163.com
 > and 126.com (and I think big.or.jp) are written in the native
 > character sets (Chinese or Japanese),

163.com and 126.com are big spam sources which (unfortuantely) also
host a few genuine users.  If you think you're getting genuine
bounces, then one of the core guys will have to help you.  If you
think it's spam-related, feel free to send copies to me (off-list) and
I can probably verify for you.  I suggest gzipped attachments as (for
obvious reasons) I have filters with big teeth for CJK spam.

------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9

[smontanaro]

    Stephen> skip@... writes:
    >> The most problematic domains are rediffmail.com, big.or.jp, 163.com,
    >> ospaz.ru and 126.com.  The bounce messages from 163.com and 126.com
    >> (and I think big.or.jp) are written in the native character sets
    >> (Chinese or Japanese),

    Stephen> 163.com and 126.com are big spam sources which (unfortuantely)
    Stephen> also host a few genuine users.

I think these are genuine bounces.  There are a few 126.com and 163.com
addresses subscribed to python-list.  The bounce messages contain actual
python-list mails in the second part of the dsn.

    Stephen> If you think it's spam-related, feel free to send copies to me
    Stephen> (off-list) and I can probably verify for you.

Just in case, I'll send one your way.

Thanks,

Skip
------------------------------------------------------
Mailman-Users mailing list
Mailman-Users@...
http://mail.python.org/mailman/listinfo/mailman-users
Mailman FAQ: http://wiki.list.org/x/AgA3
Searchable Archives: http://www.mail-archive.com/mailman-users%40python.org/
Unsubscribe: http://mail.python.org/mailman/options/mailman-users/lists%40nabble.com

Security Policy: http://wiki.list.org/x/QIA9