DCCM looping

[Rob McMahon]

OK, I've started seeing dccm looping now.  Solaris 9 (x86), compiled
with the Studio compiler.  I suspect these are the looping threads:

-----------------  lwp# 2067 / thread# 2067  --------------------
 08064cae dcc_ck_body (80f0310, 83dea08, 4214) + 146
 0805955a dccm_body (83a9c08, 83dea08, 4214) + 3a
 08061240 st_bodychunk (c363df60) + 28
 08060771 mi_engine (83a9c08) + 20d
 0805f23c mi_handle_session (83a9c08) + 30
 0805e8fe mi_thread_handle_wrapper (83a9c08) + e
 c44b4583 _thr_setup (c2432e00) + 40
 c44b4800 _lwp_start ()
-----------------  lwp# 344 / thread# 344  --------------------
 08064bc2 dcc_ck_body (80f559c, 838dd50, 7e9) + 5a
 0805955a dccm_body (8387408, 838dd50, 7e9) + 3a
 08061240 st_bodychunk (c413df60) + 28
 08060771 mi_engine (8387408) + 20d
 0805f23c mi_handle_session (8387408) + 30
 0805e8fe mi_thread_handle_wrapper (8387408) + e
 c44b4583 _thr_setup (c4490800) + 40
 c44b4800 _lwp_start ()
-----------------  lwp# 373 / thread# 373  --------------------
 08064bba dcc_ck_body (81386b8, 83c9b58, 7e9) + 52
 0805955a dccm_body (83753b0, 83c9b58, 7e9) + 3a
 08061240 st_bodychunk (c132df60) + 28
 08060771 mi_engine (83753b0) + 20d
 0805f23c mi_handle_session (83753b0) + 30
 0805e8fe mi_thread_handle_wrapper (83753b0) + e
 c44b4583 _thr_setup (c2432400) + 40
 c44b4800 _lwp_start ()

(The rest are in poll, cond_wait, sigwait, or nanosleep.)

Is there a known issue here, or am I going to have to start digging
around for the problematic messages ?  I can see that's going to be tricky.

Cheers,

Rob

--
E-Mail: Rob.McMahon@... PHONE:  [phone]
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England

_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Gary Mills]

On Wed, Jan 31, 2007 at 02:28:57PM +0000, Rob McMahon wrote:
> OK, I've started seeing dccm looping now.  Solaris 9 (x86), compiled
> with the Studio compiler.  I suspect these are the looping threads:

Same thing here with Solaris 9 SPARC.

> -----------------  lwp# 2067 / thread# 2067  --------------------
> 08064cae dcc_ck_body (80f0310, 83dea08, 4214) + 146
[..]
>
> (The rest are in poll, cond_wait, sigwait, or nanosleep.)
>
> Is there a known issue here, or am I going to have to start digging
> around for the problematic messages ?  I can see that's going to be tricky.

My current solution is to kill and restart dccm only, whenever the load
gets too high.  This seems to fix it for hours or sometimes days,
before the looping starts again.

--
-Gary Mills-    -Unix Support-    -U of M Academic Computing and Networking-
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Bruce Esquibel]

>
> On Wed, Jan 31, 2007 at 02:28:57PM +0000, Rob McMahon wrote:
> > OK, I've started seeing dccm looping now.  Solaris 9 (x86), compiled
> > with the Studio compiler.  I suspect these are the looping threads:
>
> Same thing here with Solaris 9 SPARC.


I don't want to throw anyone off the track so take this more as a comment
from the peanut gallery.

This past monday I started to migrate our mail system (which oddly enough is
all Solaris/Sparc) from the public DCC to an in-house one.

For some reason I couldn't explain it seems that DCC started to go flakey
starting about a week to ten days ago. Complaints from the end users of the
system skyrocketed, mostly from it blocking email it shouldn't and it
passing stuff that should of been blocked.

I figured going in-house would give a better track of when things go wrong,
but I'm not sure now.

I haven't noticed this looping effect with dccm but I wasn't really looking
for it either. I thought maybe there was some weird poisoning of the public
dcc servers going on (or insert your consipiracy theory here).

Four of the machines here run S10 and two others run S8, so I don't have any
S9's to check, but like I said, things seem to be humming along with the
daemons, just the effectivness and false positives seem to have gone on the
upswing since mid month compared to the two or so years we've been using it.

This is just one of those "take the above with a grain of salt" things but
wanted to pass it along.

-bruce
bje@...
 
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

I have a vague hope that an infinite loop related to another MIME
problem that I noticed and fixed late last year might solve these cases
of dccm looping.

I'm trying to get some things cleaned up so that I can make a release
today or perhaps tomorrow with that fix as well as the other recently
reported MIME problem.


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> From: Bruce Esquibel <bje@...>

> This past monday I started to migrate our mail system (which oddly enough is
> all Solaris/Sparc) from the public DCC to an in-house one.

> daemons, just the effectivness and false positives seem to have gone on the
> upswing since mid month compared to the two or so years we've been using it.

DCC effectiveness has dropped in recent months.  Perhaps that is
due to more triggering of that MIME bug.  Or perhaps not.

Clients of a DCC server that does not receive "floods" of reports of
bulk mail from the rest of the global DCC network are likely to much
less effective.  I don't recall that ripco.com is running a DCC server
connected to the global network.  In my mail logs there is some
corresondence with you about DCC servers, opus.com, Barracuda, and the
possible assignment of DCC server-IDs, but I think nothing came of it.

What DCC server are you using that is connected to the global network?
If your DCC server is not connected and you now have enough traffic
to justify a local DCC server, please contact me privately.


Note that the free license for the current DCC source does not cover
installations of DCC servers not connected to the global network.
To do that, you need to buy a commercial license.


"False positives" are different story and speak more to a misuse or
misunderstanding of distributed checksum clearinghouses.  In a sense,
albeit uninteresting, there is no such thing as a DCC false positive.
If the DCC network says that a mail message is bulk, then you can be
confident that at least one substantially identical copy of the message
has been reported before.  There two common classes of claimed DCC false
positives.  One consists of nearly empty messages consisting of few or
no words and some free mail provider advertising, shyster confidentiality
noise, or similar popular noise.  The other class consists of legitimate
bulk mail that has been detected as such, including messages from
legitimate mailing lists that has been reported to a DCC server with a
target count of "MANY." The only solution for both classes is to whitelist
such legitimate bulk mail in the system-wide /var/dcc/whiteclnt file
or appropriate per-user whiteclnt files.


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Rob McMahon]

Vernon Schryver wrote:
> I have a vague hope that an infinite loop related to another MIME
> problem that I noticed and fixed late last year might solve these cases
> of dccm looping.
>
> I'm trying to get some things cleaned up so that I can make a release
> today or perhaps tomorrow with that fix as well as the other recently
> reported MIME problem.
>
>  
We'd seen this on a few occasions running 2.3.45, with one or two
threads getting stuck.  It wasn't giving us much trouble, with a quick
restart fixing the problem for a number of days.  Until this weekend
when all hell broke loose, all email was locked solid when I came in on
Monday morning, and dccm wouldn't stay up for more than a few minutes
without turning back into a total CPU hog.  Installing 2.3.51 appears to
have fixed the problem.

So: has anyone else seen this? I'm wondering if it might have been a
deliberate DoS attack.  Do we know what qualities of a message caused
the problem?  The other alternative is that it wasn't malicious at all
but was a mass mailing of (badly formed?) email.

Cheers,

Rob

--
E-Mail: Rob.McMahon@... PHONE:  [phone]
Rob McMahon, IT Services, Warwick University, Coventry, CV4 7AL, England

_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc

[Vernon Schryver]

> From: Rob McMahon

> > I have a vague hope that an infinite loop related to another MIME
> > problem that I noticed and fixed late last year might solve these cases
> > of dccm looping.


> when all hell broke loose, all email was locked solid when I came in on
> Monday morning, and dccm wouldn't stay up for more than a few minutes
> without turning back into a total CPU hog.  Installing 2.3.51 appears to
> have fixed the problem.
>
> So: has anyone else seen this? I'm wondering if it might have been a
> deliberate DoS attack.  Do we know what qualities of a message caused
> the problem?  The other alternative is that it wasn't malicious at all
> but was a mass mailing of (badly formed?) email.

Since the problems happen on subsequent mail messages, and since a
small percentage of the millions of mail systems on the Internet use
dccm, I think the dccm looping is an unintended, even unrealized side
effect of bad spam.

For most sites, there is a bigger reason to install the current version
of the DCC code.  It significantly increases the DCC hit rates.  Some
installations that pre-filter with DNS blacklist or other mechanisms
and where most legitimate mail involves MIME have been seeing DCC spam
ratios below 20%.  Installing the current version with the other MIME
fix has generally doubled their hit rates.  Many other installations
with 30-40% hit rates with previous versions are now seeing better
than 50%.

(The ratio of spam detected by DCC clients of a DCC server to all
mail checked by the DCC clients is among the per-server graphs.)

It is necessary to install the new version on all of your DCC clients.
DCC clients are those that run dccproc, dccm, or dccifd.  Many DCC
servers have overlooked clients.  This command run as root or the
dcc user will find clients seen by a server since it was last started:
   cdcc "id XXXX; clients -V"
or
   cdcc "id XXXX; password secret; clients -V"
where "secret" is the password for server-ID XXXX in /var/dcc/ids
If you see a DCC client-serer protocol version (the 'v' column)
other than 7 such as 4 then you have a very old and creaky client.


Vernon Schryver    vjs@...
_______________________________________________
DCC mailing list      DCC@...
http://www.rhyolite.com/mailman/listinfo/dcc