zlib plugin

[Luuk-5]

hi,

Could anyone give some info on this plugin?
When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still
left with some questions...

I use Maildir to store my mails, if i enable this zlib-plugin,
should i 'manually' compress all messages?
or wil the uncompressed messages still be accessible by dovecot?
wil new messages be compressed automagically?

how should the compressed messages be named?
a message now has the name '1223224520.2783_1.opensuse:2,S'
how would a compressed message be named?
['1223224520.2783_1.opensuse:2,SZ' or '1223224520.2783_1.opensuse:2,S,Z'
or ???]

Last but not least, what gain will i have by using this plugin in
combination with Maildir, or will it have more impact when mails are
stored in mbox.

--
Luuk
$ dovecot --version
dovecot version 1.1.1
$ uname -srpmio
Linux 2.6.18.8-0.10-default x86_64 x86_64 x86_64 GNU/Linux

[Timo Sirainen]

On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:

> hi,
>
> Could anyone give some info on this plugin?
> When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still
> left with some questions...
>
> I use Maildir to store my mails, if i enable this zlib-plugin,
> should i 'manually' compress all messages?
> or wil the uncompressed messages still be accessible by dovecot?
> wil new messages be compressed automagically?

Dovecot can read both compressed and uncompressed messages.

> how should the compressed messages be named?

It doesn't matter from Dovecot's point of view since it detects the
compression by reading the first few bytes of the file. But since you'll
be compressing them manually you should mark them in some way so you
don't compress anything twice (and that would break Dovecot too).

> a message now has the name '1223224520.2783_1.opensuse:2,S'
> how would a compressed message be named?
> ['1223224520.2783_1.opensuse:2,SZ'

This is OK.

>  or '1223224520.2783_1.opensuse:2,S,Z'

This isn't a valid maildir file name.

> Last but not least, what gain will i have by using this plugin in
> combination with Maildir, or will it have more impact when mails are
> stored in mbox.

I guess it depends on the size of your mails. With mboxes it can be used
only with read-only archives, so it's not all that useful. With maildir
it can be used transparently with read-write mailboxes. I've no idea
what the performance difference or space improvements are. I guess
that'll also depend on your system.

[Luuk-5]

Timo Sirainen schreef:

> On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
>  
>> hi,
>>
>> Could anyone give some info on this plugin?
>> When i read the wiki (http://wiki.dovecot.org/Plugins/Zlib) i'm still
>> left with some questions...
>>
>> I use Maildir to store my mails, if i enable this zlib-plugin,
>> should i 'manually' compress all messages?
>> or wil the uncompressed messages still be accessible by dovecot?
>> wil new messages be compressed automagically?
>>    
>
> Dovecot can read both compressed and uncompressed messages.
>
>  
>> how should the compressed messages be named?
>>    
>
> It doesn't matter from Dovecot's point of view since it detects the
> compression by reading the first few bytes of the file. But since you'll
> be compressing them manually you should mark them in some way so you
> don't compress anything twice (and that would break Dovecot too).
>
>  
>> a message now has the name '1223224520.2783_1.opensuse:2,S'
>> how would a compressed message be named?
>> ['1223224520.2783_1.opensuse:2,SZ'
>>    
>
> This is OK.
>
>  
>>  or '1223224520.2783_1.opensuse:2,S,Z'
>>    
>
> This isn't a valid maildir file name.
>
>  
>> Last but not least, what gain will i have by using this plugin in
>> combination with Maildir, or will it have more impact when mails are
>> stored in mbox.
>>    
>
> I guess it depends on the size of your mails. With mboxes it can be used
> only with read-only archives, so it's not all that useful. With maildir
> it can be used transparently with read-write mailboxes. I've no idea
> what the performance difference or space improvements are. I guess
> that'll also depend on your system.
>  

ok, thank you,
in the near furure i will try to install this zlib-plugin (on a
test-system) and see what happens...

--
Luuk
$ dovecot --version
dovecot version 1.1.1
$ uname -srpmio
Linux 2.6.18.8-0.10-default x86_64 x86_64 x86_64 GNU/Linux

[Andrey Panin-2]

On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:

> On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
>
> > Last but not least, what gain will i have by using this plugin in
> > combination with Maildir, or will it have more impact when mails are
> > stored in mbox.
>
> I guess it depends on the size of your mails. With mboxes it can be used
> only with read-only archives, so it's not all that useful. With maildir
> it can be used transparently with read-write mailboxes. I've no idea
> what the performance difference or space improvements are. I guess
> that'll also depend on your system.

I use zlib compressed maildirs on my primary server (12000+ users) and it
achieves average compression ratio of 1.8, so if your system is io-bound
you can spend some cpu cycles to significantly decrease disk load.

--
Andrey Panin | Linux and UNIX system administrator
pazke@... | PGP key: wwwkeys.pgp.net

[Timo Sirainen]

On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:

> On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
> > On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
> >
> > > Last but not least, what gain will i have by using this plugin in
> > > combination with Maildir, or will it have more impact when mails are
> > > stored in mbox.
> >
> > I guess it depends on the size of your mails. With mboxes it can be used
> > only with read-only archives, so it's not all that useful. With maildir
> > it can be used transparently with read-write mailboxes. I've no idea
> > what the performance difference or space improvements are. I guess
> > that'll also depend on your system.
>
> I use zlib compressed maildirs on my primary server (12000+ users) and it
> achieves average compression ratio of 1.8, so if your system is io-bound
> you can spend some cpu cycles to significantly decrease disk load.

But is that 1.8 in bytes or disk blocks? Or does it make much of a
difference either way? Does 2x less space per message even matter with
small messages since most disk I/O probably goes to seeking, so reading
a single 4 kB message takes pretty much the same time as a 8 kB message?

[Curtis Maloney]

Timo Sirainen wrote:

> On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:
>> On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
>>> On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
>>>
>>>> Last but not least, what gain will i have by using this plugin in
>>>> combination with Maildir, or will it have more impact when mails are
>>>> stored in mbox.
>>> I guess it depends on the size of your mails. With mboxes it can be used
>>> only with read-only archives, so it's not all that useful. With maildir
>>> it can be used transparently with read-write mailboxes. I've no idea
>>> what the performance difference or space improvements are. I guess
>>> that'll also depend on your system.
>> I use zlib compressed maildirs on my primary server (12000+ users) and it
>> achieves average compression ratio of 1.8, so if your system is io-bound
>> you can spend some cpu cycles to significantly decrease disk load.
>
> But is that 1.8 in bytes or disk blocks? Or does it make much of a
> difference either way? Does 2x less space per message even matter with
> small messages since most disk I/O probably goes to seeking, so reading
> a single 4 kB message takes pretty much the same time as a 8 kB message?
>

Just throwing in my 0.02 currency units...

... obviously, which messages you choose to compress is up to you, and
should be tuned depending on your system.  Certainly, I wouldn't bother
compressing anything smaller than 1 allocation block (whatever that is on
your FS), and even 2, unless your CPU is really idle.

The larger the message, the greater the potential rewards.


PS: Timo... "2x smaller"? ... somehow, I expected better of you :)

--
Curtis Maloney
cmaloney@...

[Andrey Panin-2]

On 281, 10 07, 2008 at 03:10:47PM +0300, Timo Sirainen wrote:

> On Tue, 2008-10-07 at 11:38 +0400, Andrey Panin wrote:
> > On 279, 10 05, 2008 at 10:38:02PM +0300, Timo Sirainen wrote:
> > > On Sun, 2008-10-05 at 21:28 +0200, Luuk wrote:
> > >
> > > > Last but not least, what gain will i have by using this plugin in
> > > > combination with Maildir, or will it have more impact when mails are
> > > > stored in mbox.
> > >
> > > I guess it depends on the size of your mails. With mboxes it can be used
> > > only with read-only archives, so it's not all that useful. With maildir
> > > it can be used transparently with read-write mailboxes. I've no idea
> > > what the performance difference or space improvements are. I guess
> > > that'll also depend on your system.
> >
> > I use zlib compressed maildirs on my primary server (12000+ users) and it
> > achieves average compression ratio of 1.8, so if your system is io-bound
> > you can spend some cpu cycles to significantly decrease disk load.
>
> But is that 1.8 in bytes or disk blocks?

Bytes.

> Or does it make much of a
> difference either way? Does 2x less space per message even matter with
> small messages since most disk I/O probably goes to seeking, so reading
> a single 4 kB message takes pretty much the same time as a 8 kB message?

Yes, for small messages it's not very usefull, but multimegabyte mails are
quite common nowadays and for them compression is a clear win. Also I remember
that load average decreased when I started to use message compression. I have
no real numbers now, but it was noticeable.

--
Andrey Panin | Linux and UNIX system administrator
pazke@... | PGP key: wwwkeys.pgp.net