javamail, cyrus, imap:events, connection count?

[Kristian Rink-3]

Hi all;

hoping that this is the right place for this question, straight away: I
am into attaching our internal DMS to our e-mail system in order to
export / archive messages using Java-Mail. Our mail system is a cyrus
2.2 imapd, and we currently use Java Mail 1.4. My prototype solution
suffers from a rather serious flaw: As soon as I run my application, it
seems the mail server completely locks up, not responding anymore until
I restart the daemon. Looking at the server logs, this seems to be
caused by my java mail application creating wagonloads of connections
within a very short amount of time, which is peculiar as the Java class
to do this job is explicitely connecting to the store just once and, by
then, just checking whether this connection is still alive.

Pondering alternatives, there are two things I'd kindly like to ask
here:

* Is there a way to effectively limit the amount of login requests sent
to the IMAP server? As I understand [1], there is per default a pool
of one connection used, but this is not how things appear to me looking
at the mail server...

* As an alternative: I see there's the option of attaching a bunch of
listeners (of most interest in my case the MessageCountListener) to a
folder in order to deal with certain IMAP events. Tried that, also
tried setting the appropriate property (according to [1]), but haven't
seen any meaningful events / event processing goin' on. Looking at [1],
then again, I also see the IMAP events considered "highly experimental"
and, thus, wonder whether they do work at all under any circumstances?
Experience with cyrus-imapd on that, anyone?

By now, thanks for your patience and, in advance, for any possible
hints and insights.
Cheers,
Kristian


[1]http://java.sun.com/products/javamail/javadocs/com/sun/mail/imap/package-summary.html

--
Kristian Rink * http://zimmer428.net * http://flickr.com/photos/z428/
jab: kawazu@... * icq: 48874445 * fon: ++49 176 2447 2771
"One dreaming alone, it will be only a dream; many dreaming together
is the beginning of a new reality." (Hundertwasser)

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Paolo Spadafora]

*) there is a property you have to set in your properties: mail.imap.connectionpoolsize or "imaps"
            props.put("mail.imap.connectionpoolsize", "4");
            .....
            session = Session.getInstance(props, auth);
            session.setDebug(debug);
            store = (IMAPStore) session.getStore(imapProtocol);
            store.connect();

but of course you have to check your cyrus settings, it might close you connection after a certain timeout and javamail tries to re-open it.
How do you know the pool is not working? what you see in your logs?
I suggest to set to true the session debug and send the javamail logs to the list...
Anyways, if you are worried about connections and performance and have the possibility, I suggest you to put an imap proxy in front of your cyrus.

**) it works, but you have to make sure your server supports the IDLE ext and is running, use the javamail method to check:
      store.hasCapability("IDLE")
      or from the javamail logs CAPABILITY
Also take a look at the  RFC , and see what kind of output is expected and compare to the one you get in javamail.
make sure cyrus is configured properly

rpm --rebuild --with idled ......

idled
Compile the idled daemon to manage imap idle command.
Remember to start the idled daemon in /etc/cyrus.conf
 
bye

Kristian Rink ha scritto:

Hi all;

hoping that this is the right place for this question, straight away: I
am into attaching our internal DMS to our e-mail system in order to
export / archive messages using Java-Mail. Our mail system is a cyrus
2.2 imapd, and we currently use Java Mail 1.4. My prototype solution
suffers from a rather serious flaw: As soon as I run my application, it
seems the mail server completely locks up, not responding anymore until
I restart the daemon. Looking at the server logs, this seems to be
caused by my java mail application creating wagonloads of connections
within a very short amount of time, which is peculiar as the Java class
to do this job is explicitely connecting to the store just once and, by
then, just checking whether this connection is still alive. 

Pondering alternatives, there are two things I'd kindly like to ask
here:

* Is there a way to effectively limit the amount of login requests sent
to the IMAP server? As I understand [1], there is per default a pool
of one connection used, but this is not how things appear to me looking
at the mail server...

* As an alternative: I see there's the option of attaching a bunch of
listeners (of most interest in my case the MessageCountListener) to a
folder in order to deal with certain IMAP events. Tried that, also
tried setting the appropriate property (according to [1]), but haven't
seen any meaningful events / event processing goin' on. Looking at [1],
then again, I also see the IMAP events considered "highly experimental"
and, thus, wonder whether they do work at all under any circumstances?
Experience with cyrus-imapd on that, anyone?

By now, thanks for your patience and, in advance, for any possible
hints and insights.
Cheers,
Kristian


[1]http://java.sun.com/products/javamail/javadocs/com/sun/mail/imap/package-summary.html

  

=========================================================================== To unsubscribe, send email to listserv@... and include in the body of the message "signoff JAVAMAIL-INTEREST". For general help, send email to listserv@... and include in the body of the message "help".

[Kristian Rink-3]

Paolo;

first off, thank you very much for your input and comments on that.

Am Mon, 18 Feb 2008 12:55:56 +0100
schrieb Paolo Spadafora <p.spadafora@...>:

> *) there is a property you have to set in your properties:
> mail.imap.connectionpoolsize or "imaps"
>             props.put("mail.imap.connectionpoolsize", "4");
>             .....

I'll give it a try, nevertheless I thought this to default to 1, which
would be fine in my situation?


> but of course you have to check your cyrus settings, it might close
> you connection after a certain timeout and javamail tries to re-open
> it. How do you know the pool is not working? what you see in your
> logs?

In that very situation, I do iterate on a set of subfolders in a
dedicated part of the IMAP folder tree, and in my imapd log file I see
that

- the user configured to be used by the java mail application does
login something, like, 20 .. 50 times per second,

- the same moment the amount of imapd child processes created on the
mail server goes through the roof (which is surprising as the server is
configured to have 250 child processes at max - gonna check this
behaviour later).



> **) it works, but you have to make sure your server supports the IDLE
> ext and is running, use the javamail method to check:
>       store.hasCapability("IDLE")

Aaah, I see that this is the issue in my case, seems the Ubuntu/Debian
builds do have idled disabled per default. So far I don't want to
rebuild the cyrus packages, so first of all I will go for an IMAP proxy
and see how far this will get me.

Again, thank you very much for your hints.
Best regards,
Kristian

--
Kristian Rink * http://zimmer428.net * http://flickr.com/photos/z428/
jab: kawazu@... * icq: 48874445 * fon: ++49 176 2447 2771
"One dreaming alone, it will be only a dream; many dreaming together
is the beginning of a new reality." (Hundertwasser)

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Bill Shannon]

Each open folder requires a separate connection.  That's how the IMAP
protocol works.  If you're seeing lots of open connections, it's probably
because you're not closing your folders when you're done with them.

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Kristian Rink-3]

Bill, *;

first off, thanks very much for your comment and hint on that, really
much appreciated. :) Well, so far, thanks to the kind help of Paolo, I
have mainly resolved my problems in connecting to our IMAP system,
leaving one question unanswered however:


Am Mon, 18 Feb 2008 15:57:49 -0800
schrieb Bill Shannon <bill.shannon@...>:

> protocol works.  If you're seeing lots of open connections, it's
> probably because you're not closing your folders when you're done
> with them.

Actually, I did so (I surely learnt to close opened resources the
rather hard way in my early days of working with Java). That's why I
was surprised about that behaviour, as I assumed (given the
"connectionpoolsize") the application to open an amount of predefined
connections to the server and using them as they are available. But
maybe my mail server just couldn't live with having a bunch of
connections opened and closed in rather short sequence... I shall do
some more research on that the next days. :)

Anyway, thanks again for your help, have a great day everyone!
Kristian


--
Kristian Rink * http://zimmer428.net * http://flickr.com/photos/z428/
jab: kawazu@... * icq: 48874445 * fon: ++49 176 2447 2771
"One dreaming alone, it will be only a dream; many dreaming together
is the beginning of a new reality." (Hundertwasser)

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Bill Shannon]

The connectionpoolsize property controls how many connections are
ever allowed to be kept in the pool of unused connections.  Connections
are only created when a folder is opened.  They're added to the pool when
the folder is closed, if the pool isn't already full.

To see how connections enter and leave the pool, set the property
"mail.imap.connectionpool.debug" to "true".

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Kristian Rink-3]

Am Tue, 19 Feb 2008 00:02:25 -0800
schrieb Bill Shannon <bill.shannon@...>:

> Connections are only created when a folder is opened.  They're added
> to the pool when the folder is closed, if the pool isn't already full.

So the connection pool size doesn't necessarily state how many
IMAP connections in total are around? Other way 'round, having more
connections than connectionpoolsize opened is perfectly fine?

If not, having a default connectionpoolsize of 1, one would except one
(or two, given a separate store connection) connection(s) to the mail
system to be established and that connections reused as long as the app
is running...

> To see how connections enter and leave the pool, set the property
> "mail.imap.connectionpool.debug" to "true".

Thanks for pointing this out, I'll give it a try to understand things
more in-depth.

Cheers & best regards,
Kristian

--
Kristian Rink * http://zimmer428.net * http://flickr.com/photos/z428/
jab: kawazu@... * icq: 48874445 * fon: ++49 176 2447 2771
"One dreaming alone, it will be only a dream; many dreaming together
is the beginning of a new reality." (Hundertwasser)

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".

[Bill Shannon]

Kristian Rink wrote:
> So the connection pool size doesn't necessarily state how many
> IMAP connections in total are around? Other way 'round, having more
> connections than connectionpoolsize opened is perfectly fine?

Connections that are is use, because a folder is open, are not in the
pool.  The pool holds only unused connections.

> If not, having a default connectionpoolsize of 1, one would except one
> (or two, given a separate store connection) connection(s) to the mail
> system to be established and that connections reused as long as the app
> is running...

As long as you only open one folder at a time.

===========================================================================
To unsubscribe, send email to listserv@... and include in the body
of the message "signoff JAVAMAIL-INTEREST".  For general help, send email to
listserv@... and include in the body of the message "help".