Mail released from quarantine vanishing

View: New views
8 Messages — Rating Filter:   Alert me  

Mail released from quarantine vanishing

by Mark Rogers-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

I have a problem with mail released from quarantine never being delivered.

Everything had been working and nothing has changed, any suggestions
where I should look? If I search the Postfix mail logs for references to
the mail message ID I don't see anything "new" after I release the
message (ie if I see the message arrive at say 12pm, then release it at
3pm, then there's nothing logged around 3pm). The message does vanish
from the dspam user.mbox file and from the quarantine log on the web UI,
and gets shown as retrained on the web history page.

I'm stuck with 3.6.8 (Ubuntu packages).

The problem appears to be consistent, in that it seems any mail being
released goes missing.

--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555
Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG


!DSPAM:1011,486bae4b150929842711894!



Re: Mail released from quarantine vanishing

by Mark Rogers-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Mark Rogers wrote:

> I have a problem with mail released from quarantine never being
> delivered.
>
> Everything had been working and nothing has changed, any suggestions
> where I should look? If I search the Postfix mail logs for references
> to the mail message ID I don't see anything "new" after I release the
> message (ie if I see the message arrive at say 12pm, then release it
> at 3pm, then there's nothing logged around 3pm). The message does
> vanish from the dspam user.mbox file and from the quarantine log on
> the web UI, and gets shown as retrained on the web history page.
>
> I'm stuck with 3.6.8 (Ubuntu packages).
>
> The problem appears to be consistent, in that it seems any mail being
> released goes missing.
>

I'm guessing since nobody commented that nobody has any suggestions with
this? I am losing mail all over the place at the moment and would really
appreciate some suggestions as to how to debug this!

Rebooting the server tends to keep it working for a while. It *seems* to
stop working under high load (eg I tried releasing some mails from a
quarantine of 4000+ messages, the first few went through fine,
everything after that stopped working until I rebooted the server).

It seems like somehow dspam is dying but the web interface code does not
detect that, so when it sends the mail back to postfix for delivery,
maybe postfix is giving a suitable response (rejected, try again later,
whatever) but the web interface is not handling that and deleting the
unsent message from the mbox file. Suggestions as to where I would go
looking for confirmation of that theory also welcomed.

--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555
Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG


!DSPAM:1011,487389fc150924132119470!



Re: Mail released from quarantine vanishing

by Troy Ayers-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Mark Rogers wrote:

> Mark Rogers wrote:
>> I have a problem with mail released from quarantine never being
>> delivered.
>>
>> Everything had been working and nothing has changed, any suggestions
>> where I should look? If I search the Postfix mail logs for references
>> to the mail message ID I don't see anything "new" after I release the
>> message (ie if I see the message arrive at say 12pm, then release it
>> at 3pm, then there's nothing logged around 3pm). The message does
>> vanish from the dspam user.mbox file and from the quarantine log on
>> the web UI, and gets shown as retrained on the web history page.
>>
>> I'm stuck with 3.6.8 (Ubuntu packages).
>>
>> The problem appears to be consistent, in that it seems any mail being
>> released goes missing.
>>
>
> I'm guessing since nobody commented that nobody has any suggestions
> with this? I am losing mail all over the place at the moment and would
> really appreciate some suggestions as to how to debug this!
>
> Rebooting the server tends to keep it working for a while. It *seems*
> to stop working under high load (eg I tried releasing some mails from
> a quarantine of 4000+ messages, the first few went through fine,
> everything after that stopped working until I rebooted the server).
>
> It seems like somehow dspam is dying but the web interface code does
> not detect that, so when it sends the mail back to postfix for
> delivery, maybe postfix is giving a suitable response (rejected, try
> again later, whatever) but the web interface is not handling that and
> deleting the unsent message from the mbox file. Suggestions as to
> where I would go looking for confirmation of that theory also welcomed.
>
Are you using dspam as daemon?  Restart the dspam daemon only, not the
whole server.

Otherwise:

Enable debug logging.

Check for the missing emails on the root account (or it's alias) if
applicable.

Check for the missing email on the account specified to receive
double-bounces (IE postmaster)

I thought that dspam quarantine doesn't delete the message from the mbox
until after a successful exit code from whatever delivery agent was
specified.  Could somebody confirm?

Sounds like the message are not getting to postfix though.  Postfix logs
a "connect from <hostname>[ipaddress]..."  Please confirm that postfix
is listening in on the same ip/port as what you have specified in your
dspam.conf deliveryhost/deliveryport.


-Troy



!DSPAM:1011,48739350150921397012536!



Re: Mail released from quarantine vanishing

by Mark Rogers-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Troy Ayers wrote:
> Are you using dspam as daemon?  Restart the dspam daemon only, not the
> whole server.

Fair point, although the server restart is only a few seconds so that
isn't the problem, its stopping it happening in the first place!

> Otherwise:
>
> Enable debug logging.

Silly question, but where do I enable this?

I'm pretty sure that debugging isn't enabled in the Ubuntu/Debian builds
(how do I check?). I do have (and always have had) SystemLog and UserLog
enabled.

> Check for the missing emails on the root account (or it's alias) if
> applicable.
>
> Check for the missing email on the account specified to receive
> double-bounces (IE postmaster)

Where would I check where these are going?

The domain I am particularly concerned about locating missing mail from
was set as a catch-all which redirected to a single address elsewhere,
ie postmaster@ would have gone to the same place as the other emails.
Other domains which drop to POP3 boxes on my server do not, it turns
out, have postmaster aliases set up (something I need to fix).

Where would any root alias be set?

Would postfix have logged any failure to find a suitable root/postmaster
mailbox, and if so what should I look for in the logs?

> I thought that dspam quarantine doesn't delete the message from the
> mbox until after a successful exit code from whatever delivery agent
> was specified.  Could somebody confirm?

Looking at the Perl code I'm not convinced it is very careful, but I
really don't know much Perl at all. I've appended what I think is the
relevant subroutine below, but it looks to me like it extracts messages,
retrains on them (which I think leaves dspam to resend the message?),
then afterwards just deletes selected messages from the quarantine (via
Quarantine_DeleteSpam). There doesn't seem to be much attempt to avoid
deleting messages that couldn't be sent for some reason.

> Sounds like the message are not getting to postfix though.  Postfix
> logs a "connect from <hostname>[ipaddress]..."  Please confirm that
> postfix is listening in on the same ip/port as what you have specified
> in your dspam.conf deliveryhost/deliveryport.

Based on my comments above, I think you're right; maybe postfix has died
for some reason (on the port 10026 configured for dspam to send via),
and the web interface simply ignores that fact and deletes the messages.

Looking in the logs there are plenty of connections from
localhost.localdomain, but that's to be expected as the server is
usually working.

Does dspam log any failure to connect to the mailserver? I can't see it
having happened anywhere.

Next time it fails I will release something from the quarantine and
watch what goes into mail.log (and dspam's system.log).

[I assume that if I release a spam from the quarantine then retrain that
spam as spam then I get back to where I started? There's no point
releasing ham from the quarantine when I know its going to go missing!]

Thanks for your help Troy, it is very much appreciated.

Perl code from dspam.cgi follows:
sub ProcessFalsePositive {
  my(@buffer, %head, $found);
  if ($FORM{'signatureID'} eq "") {
    &error("No Message ID Specified");
  }
  open(FILE, "<$MAILBOX");
  while(<FILE>) {
    s/\r?\n$//;
    push(@buffer, $_);
  }
  close(FILE);

  while($#buffer>=0) {
    my($buff, $mode, @temp);
    $mode = 0;
    @temp = ();
    while(($buff !~ /^From /) && ($#buffer>=0)) {
      $buff = $buffer[0];
      if ($buff =~ /^From /) {
        if ($mode == 0) { $mode = 1; }
        else { next; }
      }
      $buff = shift(@buffer);
      if ($buff !~ /^From /) {
        push(@temp, $buff);
      }
      next;
    }
    foreach(@temp) {
      last if ($_ eq "");
      my($key, $val) = split(/\: ?/, $_, 2);
      $head{$key} = $val;
    }
    if ($head{'X-DSPAM-Signature'} eq $FORM{'signatureID'}) {
      $found = 1;
      open(PIPE, "|$CONFIG{'DSPAM'} $CONFIG{'DSPAM_ARGS'}  >$TMPFILE
2>&1") || &error($!);
      foreach(@temp) {
        print PIPE "$_\n";
      }
      close(PIPE);
    }
  }

  # Couldn't find the message, so just retrain on signature
  if (!$found) {
    system("$CONFIG{'DSPAM'} --source=error --class=innocent
--signature=" . quotemeta($FORM{'signatureID'}) . " --user " .
quotemeta("$CURRENT_USER"));
  }

  if ($?) {
    my(@log);
    open(LOG, "<$TMPFILE");
    @log = <LOG>;
    close(LOG);
    unlink("$TMPFILE");
    &error("<PRE>".join('', @log)."</PRE>");
  }

  unlink("$TMPFILE");
  $FORM{$FORM{'signatureID'}} = "on";
  &Quarantine_DeleteSpam();
  return;
}



--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555
Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG


!DSPAM:1011,48748205150925650348932!



Re: Mail released from quarantine vanishing

by Troy Ayers-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Mark Rogers wrote:

> Troy Ayers wrote:
>> Are you using dspam as daemon?  Restart the dspam daemon only, not
>> the whole server.
>
> Fair point, although the server restart is only a few seconds so that
> isn't the problem, its stopping it happening in the first place!
>
>> Otherwise:
>>
>> Enable debug logging.
>
> Silly question, but where do I enable this?
>
> I'm pretty sure that debugging isn't enabled in the Ubuntu/Debian
> builds (how do I check?). I do have (and always have had) SystemLog
> and UserLog enabled.
To check build parameters: dspam --version.

run dspam with the --debug switch ( my man page is a few years old,
please double check that --debug is the right switch to issue and also
check to see if it's compatible with daemon mode)

>
>> Check for the missing emails on the root account (or it's alias) if
>> applicable.
>>
>> Check for the missing email on the account specified to receive
>> double-bounces (IE postmaster)
>
> Where would I check where these are going?
Uh. the postmaster email account?
>
> The domain I am particularly concerned about locating missing mail
> from was set as a catch-all which redirected to a single address
> elsewhere, ie postmaster@ would have gone to the same place as the
> other emails. Other domains which drop to POP3 boxes on my server do
> not, it turns out, have postmaster aliases set up (something I need to
> fix).
>
> Where would any root alias be set?
Whatever your postfix specifies it to be set as.  My default location is
/etc/aliases.  The postconf utility will tell you.
>
> Would postfix have logged any failure to find a suitable
> root/postmaster mailbox, and if so what should I look for in the logs?
Yes.

You would be better served looking up postfix questions yourself, to be
sure you're getting accurate information. Something like egrep
"error|fatal|panic|warning" /var/log/maillog.

>
>> I thought that dspam quarantine doesn't delete the message from the
>> mbox until after a successful exit code from whatever delivery agent
>> was specified.  Could somebody confirm?
>
> Looking at the Perl code I'm not convinced it is very careful, but I
> really don't know much Perl at all. I've appended what I think is the
> relevant subroutine below, but it looks to me like it extracts
> messages, retrains on them (which I think leaves dspam to resend the
> message?), then afterwards just deletes selected messages from the
> quarantine (via Quarantine_DeleteSpam). There doesn't seem to be much
> attempt to avoid deleting messages that couldn't be sent for some reason.
I leave this up to those more knowledgeable than me to determine if a
bug report should be reported.

>
>> Sounds like the message are not getting to postfix though.  Postfix
>> logs a "connect from <hostname>[ipaddress]..."  Please confirm that
>> postfix is listening in on the same ip/port as what you have
>> specified in your dspam.conf deliveryhost/deliveryport.
>
> Based on my comments above, I think you're right; maybe postfix has
> died for some reason (on the port 10026 configured for dspam to send
> via), and the web interface simply ignores that fact and deletes the
> messages.
>
> Looking in the logs there are plenty of connections from
> localhost.localdomain, but that's to be expected as the server is
> usually working.
>
> Does dspam log any failure to connect to the mailserver? I can't see
> it having happened anywhere.
>
> Next time it fails I will release something from the quarantine and
> watch what goes into mail.log (and dspam's system.log).
>
> [I assume that if I release a spam from the quarantine then retrain
> that spam as spam then I get back to where I started? There's no point
> releasing ham from the quarantine when I know its going to go missing!]
Correct.

-Troy


!DSPAM:1011,4874c4dc150923992416012!



Re: Mail released from quarantine vanishing

by Mark Rogers-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Troy Ayers wrote:
> To check build parameters: dspam --version.

On the stock debian builds (well at least on the Ubuntu ones) that
doesn't give any output. Never really understood why.

> run dspam with the --debug switch ( my man page is a few years old,
> please double check that --debug is the right switch to issue and also
> check to see if it's compatible with daemon mode)

I'll see what I can find out, but I think dspam needs building with
--enable-debug for that to do anything useful, and I think (based on the
advice in dspam.conf) that is not how it was built.

>> Where would any root alias be set?
> Whatever your postfix specifies it to be set as.  My default location
> is /etc/aliases.  The postconf utility will tell you.

OK, postmaster is aliased to root, and root to user1 (the main user I
log into the server with, which isn't really user1 in case anyone wants
to try and hack my server :-)

I have no idea where that mail actually goes to, however. If I try:
    $mail user1
.. to send an email to user1, then:
    mail user1
.. I get "No mail for user1". There is nothing in Postfix's mail.log
which refers to user1.

I have modified the alias to go to a full email address that should work.

>>
>> Would postfix have logged any failure to find a suitable
>> root/postmaster mailbox, and if so what should I look for in the logs?
> Yes.
>
> You would be better served looking up postfix questions yourself, to
> be sure you're getting accurate information. Something like egrep
> "error|fatal|panic|warning" /var/log/maillog.

Thanks, and yes I understand that this is not a Postfix list and will
check out any advice I get here separately. (Aside from the log file
being mail.log it looks pretty sound though.)

The only dspam related errors seem to be lots of:
     process_message returned error -5.  dropping message.

.. although I have no easy way to know if they're related to my problem.
Any idea what this means?

>> Looking at the Perl code I'm not convinced it is very careful, but I
>> really don't know much Perl at all. I've appended what I think is the
>> relevant subroutine below, but it looks to me like it extracts
>> messages, retrains on them (which I think leaves dspam to resend the
>> message?), then afterwards just deletes selected messages from the
>> quarantine (via Quarantine_DeleteSpam). There doesn't seem to be much
>> attempt to avoid deleting messages that couldn't be sent for some
>> reason.
> I leave this up to those more knowledgeable than me to determine if a
> bug report should be reported.

I certainly hope someone is able to look at the code, as if there is a
problem it'll be affecting more than just me, even if not always visibly!

If someone is at least able to compare the code I posted with the same
code in 3.8.x that might be useful.

Thanks again for your help.

--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555
Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG


!DSPAM:1011,4874d683150923427311555!



Re: Mail released from quarantine vanishing

by Troy Ayers-2 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Mark Rogers wrote:

> Troy Ayers wrote:
>> To check build parameters: dspam --version.
>
> On the stock debian builds (well at least on the Ubuntu ones) that
> doesn't give any output. Never really understood why.
>
>> run dspam with the --debug switch ( my man page is a few years old,
>> please double check that --debug is the right switch to issue and
>> also check to see if it's compatible with daemon mode)
>
> I'll see what I can find out, but I think dspam needs building with
> --enable-debug for that to do anything useful, and I think (based on
> the advice in dspam.conf) that is not how it was built.
>
>>> Where would any root alias be set?
>> Whatever your postfix specifies it to be set as.  My default location
>> is /etc/aliases.  The postconf utility will tell you.
>
> OK, postmaster is aliased to root, and root to user1 (the main user I
> log into the server with, which isn't really user1 in case anyone
> wants to try and hack my server :-)
>
> I have no idea where that mail actually goes to, however. If I try:
>    $mail user1
> . to send an email to user1, then:
>    mail user1
> . I get "No mail for user1". There is nothing in Postfix's mail.log
> which refers to user1.
>
> I have modified the alias to go to a full email address that should work.
>
>>>
>>> Would postfix have logged any failure to find a suitable
>>> root/postmaster mailbox, and if so what should I look for in the logs?
>> Yes.
>>
>> You would be better served looking up postfix questions yourself, to
>> be sure you're getting accurate information. Something like egrep
>> "error|fatal|panic|warning" /var/log/maillog.
>
> Thanks, and yes I understand that this is not a Postfix list and will
> check out any advice I get here separately. (Aside from the log file
> being mail.log it looks pretty sound though.)
>
> The only dspam related errors seem to be lots of:
>     process_message returned error -5.  dropping message.
>
> . although I have no easy way to know if they're related to my
> problem. Any idea what this means?
I think process_message only has to do with (re)training, so this would
indicate a training failure only?  There should be more logging just
before that, which may be helpful in that determination.
 -Troy


!DSPAM:1011,4874fb29150923415841507!



Re: Mail released from quarantine vanishing

by Mark Rogers-5 :: Rate this Message:

Reply to Author | View Threaded | Show Only this Message

Troy Ayers wrote:
> I think process_message only has to do with (re)training, so this
> would indicate a training failure only?  There should be more logging
> just before that, which may be helpful in that determination.

Looking further into the logs, the process_message error occurs as follows:
> Jul 10 12:35:03 moresaa5 dspam[5026]: Unable to find a valid
> signature. Aborting.
> Jul 10 12:35:03 moresaa5 dspam[5026]: process_message returned error
> -5.  dropping message.

I'm not sure under what circumstances dspam would be unable to find a
valid signature, but there does *seem* to be some correlation between
the problem happening and the times that these events occur in the logs.

Can anyone confirm whether these errors are routine or not? My mail
server (which is not particularly busy) has 73 of these errors in
yesterday's log. A quick look at the log says that's against 10000
emails of which 7000 were classified as spam.

--
Mark Rogers // More Solutions Ltd (Peterborough Office) // 0845 45 89 555
Registered in England (0456 0902) at 13 Clarke Rd, Milton Keynes, MK1 1LG


!DSPAM:1011,4875ffbe150928051213871!


LightInTheBox - Buy quality products at wholesale price!