Strange conversations with the server

[chubbard]

I'm having a hell of a time with javamail retrieving mailboxes over
POP3.  It works, but it's very inconsistent.  I believe the problem is
that I have a read timeout happen which throws a MessagingException.
And, in my finally block I try to close the folder and mailbox to
cleanup appropriately.  The client tries to send a series of deletes
to the server, at which time it wakes up from it's timeout and tries
to send the response back to me.  Javamail freaks out because it's
getting something it doesn't expect.

I see this also happen with OutOfMemoryErrors too where by trying to
cleanly shutdown after one of these is impossible because the client
freaks out and the server is still sending the message to me.  I'd
like to handle this better, but I'm not sure I can given the
limitations of javamail API.  Like I'd like to save the mail straight
tto disk without creating any memory.  Then do my processing on the
disk copy rather than in memory.  That way javamail wouldn't be
affected by the client running out of memory.

Both of these problems really affect my program badly because I'm
using POP whereby I can't delete mails from the server without closing
the mailbox so I've tried to work around these problems by only
processing batches of emails then closing the connection so I don't
have to start over if I encounter an error.  These errors lead to
duplicates being generated because I encounter emails more than once.

Here is my mail.debug output of the problem:

2008-06-24 09:46:13,123 [Mail loader] DEBUG
com.emailarchive.demon.MailDemonLoader  - Downloading message took 0ms
C: TOP 85 0
S: +OK Reading Message
Return-Path: <bhinbest@...>
Received: from FINST11 (unknown [10.1.20.3])
    by scalix.maplelodgefarms.com (Scalix SMTP Relay 11.3.0.11339)
    via ESMTP; Mon, 23 Jun 2008 16:13:55 -0400 (EDT)
Date: Mon, 23 Jun 2008 16:13:20 -0400
From: "Brent Hinbest" <bhinbest@...>
To: "'Debbie Carey'" <Debbie.Carey@...>,
        "'Tyson Hall/Toronto/IBM'" <Tyson.Hall@...>
cc: "'Scott Evans'" <sevans@...>
Message-ID: <!&!AAAAAAAAAAAYAAAAAAAAAB/57uhFZjNIkhNLRYLn3trCgAAAEAAAACzj6aD85xJA
iwMtkC3iriEBAAAAAA==@...>
Subject: TM1 Data
Disposition-Notification-To: "Brent Hinbest" <bhinbest@...>
Return-Receipt-To: "Brent Hinbest" <bhinbest@...>
X-Mailer: Microsoft Office Outlook 11
Thread-Index: AcjVbZMH1XrmQT39T3ytcOBW5DGjOw==
X-MimeOLE: Produced By Microsoft MimeOLE V6.00.2900.3198
MIME-Version: 1.0
Content-Type: multipart/mixed;
        boundary="----=_NextPart_000_0039_01C8D54C.0E21DC30"

.
2008-06-24 09:46:13,165 [Mail loader] DEBUG
com.emailarchive.demon.MailIndexer  - Processing
<!&!AAAAAAAAAAAYAAAAAAAAAB/57uhFZjNIkhNLRYLn3trCgAAAEAAAACzj6aD85xJAiwMtkC3iriEBAAAAAA==@...>
C: RETR 85
2008-06-24 09:46:43,291 [Mail loader] DEBUG
com.emailarchive.demon.MailDemonLoader  - Closing folder: INBOX
C: DELE 1
S: +OK Reading Message
C: DELE 2
S: Return-Path: <bhinbest@...>
2008-06-24 09:46:59,339 [Mail loader] DEBUG
com.emailarchive.demon.MailDemonLoader  -
javax.mail.MessagingException: Exception deleting messages during
close;
  nested exception is:
        java.io.IOException: Unexpected response: Return-Path:
<bhinbest@...>
2008-06-24 09:46:59,339 [Mail loader] DEBUG
com.emailarchive.demon.MailDemonLoader  - Closing mailbox.
2008-06-24 09:46:59,339 [Mail loader] ERROR
com.emailarchive.demon.MailDemonLoader  - error fetching POP3 content
javax.mail.MessagingException: error fetching POP3 content;
  nested exception is:
        java.net.SocketTimeoutException: Read timed out
        at com.sun.mail.pop3.POP3Message.getContentStream(POP3Message.java:199)
        at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1701)
        at javax.mail.internet.MimeMessage.writeTo(MimeMessage.java:1659)
        at com.emailarchive.entity.MessageEntity.save(MessageEntity.java:296)
        at com.emailarchive.entity.MessageEntity.parseMimeMessage(MessageEntity.java:241)

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Bill Shannon]

Charlie Hubbard wrote:
> I'm having a hell of a time with javamail retrieving mailboxes over
> POP3.  It works, but it's very inconsistent.  I believe the problem is
> that I have a read timeout happen which throws a MessagingException.
> And, in my finally block I try to close the folder and mailbox to
> cleanup appropriately.  The client tries to send a series of deletes
> to the server, at which time it wakes up from it's timeout and tries
> to send the response back to me.  Javamail freaks out because it's
> getting something it doesn't expect.

Interesting problem.  I think I've been assuming that when a timeout
occurs, it's because the server is dead, and further attempts to use
the connection will also timeout or fail.  Obviously that's not
necessarily true.

I suppose the POP3 provider ought to catch the timeouts and either
close the connection immediately or mark it as "dead" so that the
eventual close of the folder doesn't try to send the delete commands.

> I see this also happen with OutOfMemoryErrors too where by trying to
> cleanly shutdown after one of these is impossible because the client
> freaks out and the server is still sending the message to me.  I'd
> like to handle this better, but I'm not sure I can given the
> limitations of javamail API.  Like I'd like to save the mail straight
> tto disk without creating any memory.  Then do my processing on the
> disk copy rather than in memory.  That way javamail wouldn't be
> affected by the client running out of memory.

Running out of memory is harder.  There's very little you can do in
Java that doesn't require allocating memory.  Possibly if you've opened
the file and ensured that all buffers are allocated, you might be able
to write the message to the file.  Possibly.  If you're lucky and JavaMail
doesn't need to allocate any memory to write out the message.

> Both of these problems really affect my program badly because I'm
> using POP whereby I can't delete mails from the server without closing
> the mailbox so I've tried to work around these problems by only
> processing batches of emails then closing the connection so I don't
> have to start over if I encounter an error.  These errors lead to
> duplicates being generated because I encounter emails more than once.

I think I can improve the first problem, but I'm not sure there's anything
reasonable that can be done for the second problem.

Do you want to test any improvements for the first problem?

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[chubbard]

Sure I'll be happy to test any changes.  Skipping the deletes when an
exception happens might make the API behave a little nicer, but it
still sticks me with a big problem that it is very hard to solve with
javamail.  Say I'm parsing 1000 messages, and deleting them as I go,
and then I get to the 999th message and the server pauses.  Bam
exception.  Now the next time I talk back to that mailbox I get all
999 messages over again.  If I could send deletes immediately that
would really help instead of at the end.

I think there is more that can be done for the second problem though.
Generally, I'm working with large mailboxes (order of 10,000 messages
sometimes with 100MB attachments -- don't ask me I don't manage their
servers).  The problem with the POP provider is it allocates a buffer
the size of the mailbox regardless of folder.getMessage() API I use.
I've tried to work around this by dropping the cache of emails with
the following:

        mimeMessage.setFlag(Flags.Flag.DELETED, true);
        if( mimeMessage instanceof POP3Message ) {
            ((POP3Message)mimeMessage).invalidate(true);
        }

The idea is that it would get rid of the memory occupied by
mimeMessage, but I'm not entirely sure it's getting rid of enough.
I'm giving 512MB to my JVM, and it still sometimes runs out of memory.
 I've profiled my application and all of the memory is being taken up
by MimeMessages.    Either a way to truly bypass caching or institute
a page scheme so I can limit the amount of memory needed for reading
the store, or a way to stream the socket's response directly to a
stream without parsing the email into memory would be optimal.  Then I
can use javamail to parse that file on the disk one by one.  Making
sure that I only use memory for one message at a time.  The last
option might be simpler to implement from your standpoint.  Bypassing
the cache is very similar to the last option.

Thanks
Charlie

On Tue, Jun 24, 2008 at 10:00 PM, Bill Shannon <bill.shannon@...> wrote:

> Charlie Hubbard wrote:
>>
>> I'm having a hell of a time with javamail retrieving mailboxes over
>> POP3.  It works, but it's very inconsistent.  I believe the problem is
>> that I have a read timeout happen which throws a MessagingException.
>> And, in my finally block I try to close the folder and mailbox to
>> cleanup appropriately.  The client tries to send a series of deletes
>> to the server, at which time it wakes up from it's timeout and tries
>> to send the response back to me.  Javamail freaks out because it's
>> getting something it doesn't expect.
>
> Interesting problem.  I think I've been assuming that when a timeout
> occurs, it's because the server is dead, and further attempts to use
> the connection will also timeout or fail.  Obviously that's not
> necessarily true.
>
> I suppose the POP3 provider ought to catch the timeouts and either
> close the connection immediately or mark it as "dead" so that the
> eventual close of the folder doesn't try to send the delete commands.
>
>> I see this also happen with OutOfMemoryErrors too where by trying to
>> cleanly shutdown after one of these is impossible because the client
>> freaks out and the server is still sending the message to me.  I'd
>> like to handle this better, but I'm not sure I can given the
>> limitations of javamail API.  Like I'd like to save the mail straight
>> tto disk without creating any memory.  Then do my processing on the
>> disk copy rather than in memory.  That way javamail wouldn't be
>> affected by the client running out of memory.
>
> Running out of memory is harder.  There's very little you can do in
> Java that doesn't require allocating memory.  Possibly if you've opened
> the file and ensured that all buffers are allocated, you might be able
> to write the message to the file.  Possibly.  If you're lucky and JavaMail
> doesn't need to allocate any memory to write out the message.
>
>> Both of these problems really affect my program badly because I'm
>> using POP whereby I can't delete mails from the server without closing
>> the mailbox so I've tried to work around these problems by only
>> processing batches of emails then closing the connection so I don't
>> have to start over if I encounter an error.  These errors lead to
>> duplicates being generated because I encounter emails more than once.
>
> I think I can improve the first problem, but I'm not sure there's anything
> reasonable that can be done for the second problem.
>
> Do you want to test any improvements for the first problem?
>

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Bill Shannon]

Charlie Hubbard wrote:
> Sure I'll be happy to test any changes.  Skipping the deletes when an
> exception happens might make the API behave a little nicer, but it
> still sticks me with a big problem that it is very hard to solve with
> javamail.  Say I'm parsing 1000 messages, and deleting them as I go,
> and then I get to the 999th message and the server pauses.  Bam
> exception.  Now the next time I talk back to that mailbox I get all
> 999 messages over again.  If I could send deletes immediately that
> would really help instead of at the end.

The server is only supposed to process the DELE command if the connection
is closed normally.  From RFC 1939:

       DELE msg
...
          Discussion:
              The POP3 server marks the message as deleted.  Any future
              reference to the message-number associated with the message
              in a POP3 command generates an error.  The POP3 server does
              not actually delete the message until the POP3 session
              enters the UPDATE state.
...
    When the client issues the QUIT command from the TRANSACTION state,
    the POP3 session enters the UPDATE state.  (Note that if the client
    issues the QUIT command from the AUTHORIZATION state, the POP3
    session terminates but does NOT enter the UPDATE state.)

    If a session terminates for some reason other than a client-issued
    QUIT command, the POP3 session does NOT enter the UPDATE state and
    MUST not remove any messages from the maildrop.

Issuing the DELE command at the beginning of the session isn't going
to help you.

Really, you're stuck with keeping track of which messages you've processed
and recovering from any failures when you reconnect.

> I think there is more that can be done for the second problem though.
> Generally, I'm working with large mailboxes (order of 10,000 messages
> sometimes with 100MB attachments -- don't ask me I don't manage their
> servers).  The problem with the POP provider is it allocates a buffer
> the size of the mailbox regardless of folder.getMessage() API I use.
> I've tried to work around this by dropping the cache of emails with
> the following:
>
>         mimeMessage.setFlag(Flags.Flag.DELETED, true);
>         if( mimeMessage instanceof POP3Message ) {
>             ((POP3Message)mimeMessage).invalidate(true);
>         }
>
> The idea is that it would get rid of the memory occupied by
> mimeMessage, but I'm not entirely sure it's getting rid of enough.

If you have evidence that it's not, let me know.

> I'm giving 512MB to my JVM, and it still sometimes runs out of memory.
>  I've profiled my application and all of the memory is being taken up
> by MimeMessages.    Either a way to truly bypass caching or institute
> a page scheme so I can limit the amount of memory needed for reading
> the store, or a way to stream the socket's response directly to a
> stream without parsing the email into memory would be optimal.  Then I
> can use javamail to parse that file on the disk one by one.  Making
> sure that I only use memory for one message at a time.  The last
> option might be simpler to implement from your standpoint.  Bypassing
> the cache is very similar to the last option.

While it's hard to recover from running out of memory, you're right
that there are things that can be done to reduce the amount of memory
used.  I know how to do a more efficient buffering scheme, but I've just
never had time to implement it.

I'd also like to implement a scheme that caches POP3 messages in files
rather than in memory.  Again, I know what to do, I just haven't had
time to do it.

For your use, making sure that invalidate is having the intended effect
is probably a good place to start.

Here's another thing to try...  Before fetching any message content,
call Message.getSize().  This will cause the POP3 provider to discover
the size of the message, which it will use later when fetching the
message content to preallocate the buffer for the message, avoiding
the need for ByteArrayOutputStream to expand the buffer as data is
read.  Let me know if that helps.

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".