Releasing 2.1.12 - a new release manager?

[Carsten Ziegeler]

Hi,

I think it's really time now to get 2.1.12 out of the door.
As I'm not involved in Cocoon anymore I would like to step down as the
release manager; imho it makes more sense if someone still working
on/with Cocoon does this job :)
It's really easy, everything is outlined in our docs and I can help
during the process if required. So any volunteers?

Carsten
--
Carsten Ziegeler
cziegeler@...

[Joerg Heinicke]

On 28.05.2008 02:05, Carsten Ziegeler wrote:

> I think it's really time now to get 2.1.12 out of the door.
> As I'm not involved in Cocoon anymore I would like to step down as the
> release manager; imho it makes more sense if someone still working
> on/with Cocoon does this job :)
> It's really easy, everything is outlined in our docs and I can help
> during the process if required. So any volunteers?

If it is only 2.1 (at least for the time being) I can do this job
(though I'm neither working with Cocoon anymore lately). Just have to
check this documentation (any links?) what it needs to do such a release
(signed PGP key or things like that).

Only issue I want to solve before the release is the
BufferedOutputStream issue. I planned to do it this weekend.

Joerg

[Carsten Ziegeler]

Joerg Heinicke wrote:
> On 28.05.2008 02:05, Carsten Ziegeler wrote:
>
> If it is only 2.1 (at least for the time being) I can do this job
> (though I'm neither working with Cocoon anymore lately). Just have to
> check this documentation (any links?) what it needs to do such a release
> (signed PGP key or things like that).
Here is the whole "guide" :)
http://wiki.apache.org/cocoon/CocoonReleaseHowTo

You need a key which you have to put up on our site into the KEYS file.

Now, I can do the release - although I think it makes more sense someone
more close to the project than I am should do the work. Whatever you prefer.

> Only issue I want to solve before the release is the
> BufferedOutputStream issue. I planned to do it this weekend.
Great

Carsten

--
Carsten Ziegeler
cziegeler@...

[Joerg Heinicke]

On 29.05.2008 00:06, Joerg Heinicke wrote:

> Only issue I want to solve before the release is the
> BufferedOutputStream issue. I planned to do it this weekend.

Done. Please review the file attached. It's still completely untested.
At the moment I need some sleep ;) I will write junit tests for it this
week and eventually commit it.

Joerg

/*
 * Licensed to the Apache Software Foundation (ASF) under one or more
 * contributor license agreements.  See the NOTICE file distributed with
 * this work for additional information regarding copyright ownership.
 * The ASF licenses this file to You under the Apache License, Version 2.0
 * (the "License"); you may not use this file except in compliance with
 * the License.  You may obtain a copy of the License at
 *
 *      http://www.apache.org/licenses/LICENSE-2.0
 *
 * Unless required by applicable law or agreed to in writing, software
 * distributed under the License is distributed on an "AS IS" BASIS,
 * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 * See the License for the specific language governing permissions and
 * limitations under the License.
 */
package org.apache.cocoon.util;

import java.io.FilterOutputStream;
import java.io.IOException;
import java.io.OutputStream;

/**
 * This class is similar to the {@link java.io.BufferedOutputStream}. In
 * addition it provides an increasing buffer, the possibility to reset the
 * buffer and it counts the number of bytes written to the output stream.
 *
 * @author <a href="mailto:cziegeler@...">Carsten Ziegeler</a>
 * @version CVS $Id: BufferedOutputStream.java 433543 2006-08-22 06:22:54Z crossley $
 * @since   2.1
 */
public class BufferedOutputStream extends FilterOutputStream {

    private byte buffer[];

    private int count;
    private int totalCount;

    private final int flushBufferSize;

    /**
     * Creates a new buffered output stream to write data to the
     * specified underlying output stream with a default 8192-byte
     * buffer size.
     *
     * @param   out   the underlying output stream.
     */
    public BufferedOutputStream(final OutputStream out) {
        this(out, 8192, 32768);
    }

    /**
     * Creates a new buffered output stream to write data to the specified
     * underlying output stream with the specified buffer sizes.
     *
     * @param out    the underlying output stream.
     * @param initialBufferSize  the initial buffer size. Must be greater than 0.
     *                           Will be limited to the flush buffer size.
     * @param flushBufferSize  the buffer size when the stream is flushed. Must
     *                         be greater than 0 or -1 meaning the stream never
     *                         flushes itself.
     */
    public BufferedOutputStream(final OutputStream out,
                                final int initialBufferSize,
                                final int flushBufferSize) {
        super(out);
        if (initialBufferSize <= 0) {
            throw new IllegalArgumentException("Initial buffer size <= 0");
        }
        if (flushBufferSize <= 0 && flushBufferSize != -1) {
            throw new IllegalArgumentException("Flush buffer size <= 0 && != -1");
        }
        this.buffer =
            new byte[initialBufferSize > flushBufferSize ? flushBufferSize : initialBufferSize];
        this.flushBufferSize = flushBufferSize;
    }

    /**
     * Writes the specified byte to this buffered output stream.
     *
     * @param      b   the byte to be written.
     * @exception  IOException  if an I/O error occurs.
     */
    public void write(int b) throws IOException {
        if (this.count == this.buffer.length) {
            // No need to check return value, can NEVER be 0.
            this.increaseBuffer(1);
        }

        this.buffer[this.count++] = (byte)b;
        this.totalCount++;

        if (this.count == this.flushBufferSize) {
            flush();
        }
    }

    /**
     * Writes <code>len</code> bytes from the specified byte array
     * starting at offset <code>off</code> to this buffered output stream.
     *
     * <p> Ordinarily this method stores bytes from the given array into this
     * stream's buffer, flushing the buffer to the underlying output stream as
     * needed.  If the requested length is at least as large as this stream's
     * buffer, however, then this method will flush the buffer and write the
     * bytes directly to the underlying output stream.  Thus redundant
     * <code>BufferedOutputStream</code>s will not copy data unnecessarily.
     *
     * @param      b     the data.
     * @param      off   the start offset in the data.
     * @param      len   the number of bytes to write.
     * @exception  IOException  if an I/O error occurs.
     */
    public void write(final byte[] b, final int off, final int len) throws IOException {
        if (len > this.buffer.length - this.count) {
            int actualIncrease = this.increaseBuffer(len);
            if (actualIncrease < len) {
                // Needs to be written in chunks by recursive calls to this method.
                write(b, off, actualIncrease);
                write(b, off + actualIncrease, len - actualIncrease);
                return;
            }
        }

        System.arraycopy(b, off, this.buffer, this.count, len);
        this.count += len;
        this.totalCount += len;

        if (this.count == this.flushBufferSize) {
            flush();
        }
    }

    /**
     * Flushes this buffered output stream.
     *
     * @exception  IOException  if an I/O error occurs.
     */
    public void flush() throws IOException {
        writeBuffer();
        this.out.flush();
    }

    /**
     * Closes this buffered output stream.
     * Flush before closing.
     *
     * @exception  IOException  if an I/O error occurs.
     */
    public void close() throws IOException {
        flush();
        super.close();
    }

    /**
     * Write the buffer
     */
    private void writeBuffer() throws IOException {
        if (this.count > 0) {
            this.out.write(this.buffer, 0, this.count);
            this.clearBuffer();
        }
    }

    /**
     * Increase the buffer by at least as many bytes as specified via the
     * parameter, but not exceeding the flushBufferSize. The actual increase is
     * returned.
     *
     * @return  increase in buffer size.
     */
    private int increaseBuffer(final int increase) {
        int oldLength = this.buffer.length;
        if (oldLength == this.flushBufferSize) {
            // fast way out
            return 0;
        }

        int newLength = oldLength;
        int actualIncrease;
        do {
            newLength = newLength * 2;
            if (this.flushBufferSize > 0 && newLength >= this.flushBufferSize) {
                newLength = this.flushBufferSize;
                actualIncrease = newLength - oldLength;
                break;
            }
            actualIncrease = newLength - oldLength;
        } while (actualIncrease < increase);

        // Because of the "fast way out" above at this point there should always be an increase.
        byte[] newBuffer = new byte[newLength];
        if (this.count > 0) {
            System.arraycopy(this.buffer, 0, newBuffer, 0, this.count);
        }
        this.buffer = newBuffer;
        return actualIncrease;
    }

    /**
     * Clear/reset the buffer
     * @deprecated Public access is deprecated. Use {@link #reset()} instead.
     */
    public void clearBuffer() {
        this.count = 0;
    }

    /**
     * Reset the BufferedOutputStream to the last {@link #flush()}.
     */
    public void reset() {
        this.clearBuffer();
    }

    /**
     * Return the size of the current buffer
     */
    public int getCount() {
        return this.totalCount;
    }

}

[Sylvain Wallez]

Joerg Heinicke wrote:
> On 29.05.2008 00:06, Joerg Heinicke wrote:
>
>> Only issue I want to solve before the release is the
>> BufferedOutputStream issue. I planned to do it this weekend.
>
> Done. Please review the file attached. It's still completely untested.
> At the moment I need some sleep ;) I will write junit tests for it
> this week and eventually commit it.

Stupid question: why do we need a special BufferedOutputStream?

Sylvain

--
Sylvain Wallez - http://bluxte.net

[Joerg Heinicke]

On 02.06.2008 05:56, Sylvain Wallez wrote:

>>> Only issue I want to solve before the release is the
>>> BufferedOutputStream issue. I planned to do it this weekend.
>>
>> Done. Please review the file attached. It's still completely untested.
>> At the moment I need some sleep ;) I will write junit tests for it
>> this week and eventually commit it.
>
> Stupid question: why do we need a special BufferedOutputStream?

For being able to reset the response buffer for error handling. This is
also possible with java.io.BufferedOutputStream, if the buffer size is
big enough (current default value is 1MB), but then the buffer byte[] is
always that big rather than increasing. That's what's happening right
now, if you don't specify -1 as buffer size. -1 means complete buffering
which on the other hand might lead to OutOfMemoryError [1]. In addition
our BOS counts the bytes so that we can use the value to set the content
length header.

Joerg

[1] https://issues.apache.org/jira/browse/COCOON-2168

[Reinhard Pötz-2]

Joerg Heinicke wrote:

> On 02.06.2008 05:56, Sylvain Wallez wrote:
>
>>>> Only issue I want to solve before the release is the
>>>> BufferedOutputStream issue. I planned to do it this weekend.
>>>
>>> Done. Please review the file attached. It's still completely
>>> untested. At the moment I need some sleep ;) I will write junit tests
>>> for it this week and eventually commit it.
>>
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This is
> also possible with java.io.BufferedOutputStream, if the buffer size is
> big enough (current default value is 1MB), but then the buffer byte[] is
> always that big rather than increasing. That's what's happening right
> now, if you don't specify -1 as buffer size. -1 means complete buffering
> which on the other hand might lead to OutOfMemoryError [1]. In addition
> our BOS counts the bytes so that we can use the value to set the content
> length header.

Is my understanding right that the content length header can only be set
as long as you haven't written into the underlying output stream?

--
Reinhard Pötz                           Managing Director, {Indoqa} GmbH
                          http://www.indoqa.com/en/people/reinhard.poetz/

Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member                  reinhard@...
________________________________________________________________________

[Joerg Heinicke]

Reinhard Pötz <reinhard <at> apache.org> writes:

> Is my understanding right that the content length header can only be set
> as long as you haven't written into the underlying output stream?

Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
have been flushed yet.

Joerg

[Sylvain Wallez]

Joerg Heinicke wrote:

> On 02.06.2008 05:56, Sylvain Wallez wrote:
>
>>>> Only issue I want to solve before the release is the
>>>> BufferedOutputStream issue. I planned to do it this weekend.
>>>
>>> Done. Please review the file attached. It's still completely
>>> untested. At the moment I need some sleep ;) I will write junit
>>> tests for it this week and eventually commit it.
>>
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This
> is also possible with java.io.BufferedOutputStream, if the buffer size
> is big enough (current default value is 1MB), but then the buffer
> byte[] is always that big rather than increasing. That's what's
> happening right now, if you don't specify -1 as buffer size. -1 means
> complete buffering which on the other hand might lead to
> OutOfMemoryError [1]. In addition our BOS counts the bytes so that we
> can use the value to set the content length header.

Got it. I'm using a similar technique with a servlet filter providing
the response with a ByteArrayOutputStream that gives the size and has
the convenient writeTo(OutputStream) method. Now in my case [1] pages
are always small and I don't need to put a high limit on the size of the
buffer.

Sylvain

[1] http://bluxte.net/blog/2008-04/29-54-29.html

--
Sylvain Wallez - http://bluxte.net

[Reinhard Pötz-2]

Joerg Heinicke wrote:
> Reinhard Pötz <reinhard <at> apache.org> writes:
>
>> Is my understanding right that the content length header can only be set
>> as long as you haven't written into the underlying output stream?
>
> Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
> have been flushed yet.

What happens when the buffer size is exceeded? Does it mean that the
output is streamed and the content length parameter can't be set?

--
Reinhard Pötz                           Managing Director, {Indoqa} GmbH
                          http://www.indoqa.com/en/people/reinhard.poetz/

Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member                  reinhard@...
________________________________________________________________________

[Joerg Heinicke]

Reinhard Pötz <reinhard <at> apache.org> writes:

> >> Is my understanding right that the content length header can only be set
> >> as long as you haven't written into the underlying output stream?
> >
> > Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
> > have been flushed yet.
>
> What happens when the buffer size is exceeded? Does it mean that the
> output is streamed and the content length parameter can't be set?

If the buffer size is exceeded the output stream is flushed and eventually
written to the response. A header like the content-length can no longer be set.

That's why Cocoon's BufferedOutputStream was setup to buffer all and everything
by default which lead to COCOON-2168 [1] with Cocoon's default setting. With the
change to a buffer of 1 MB rather than endless buffering we switched back to
java.io.BufferedOutputStream, but this allocates the memory for the buffer
immediately. So even with 10 KB pages you always end up with 1 MB byte[]. I
intend to fix this in our BufferedOutputStream with an increasing buffer and the
two configuration parameters initial buffer size and flush buffer size.

Just to make it clear: We talk about default configuration of Cocoon here. You
still can enforce complete buffering by setting flush buffer size to -1 again.
But it's up to the user to enforce this and potentially run into OOME. It's also
up to the users to find a reasonable flush buffer size as it always used to be
when they haven't used the endless buffering. Only thing that is going to change
is in case of not endless buffering: The full buffer is not allocated
immediately but step by step.

Joerg

[1] https://issues.apache.org/jira/browse/COCOON-2168

[Andy Stevens-2]

Joerg Heinicke wrote:

> On 02.06.2008 05:56, Sylvain Wallez wrote:
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This is
> also possible with java.io.BufferedOutputStream, if the buffer size is
> big enough (current default value is 1MB), but then the buffer byte[] is
> always that big rather than increasing. That's what's happening right
> now, if you don't specify -1 as buffer size. -1 means complete buffering
> which on the other hand might lead to OutOfMemoryError [1]. In addition
> our BOS counts the bytes so that we can use the value to set the content
> length header.
>
> Joerg
>
> [1] https://issues.apache.org/jira/browse/COCOON-2168

Out of interest, have you raised an enhancement request and/or a patch
with the OpenJDK project[1]?  It seems to me this "increasing buffer"
behaviour would be useful more generally...


Andy.

[1] http://openjdk.java.net/
--
http://pseudoq.sourceforge.net/  Open source Sudoku solver

[Joerg Heinicke]

On 07.06.2008 03:13, Andy Stevens wrote:

> Out of interest, have you raised an enhancement request and/or a patch
> with the OpenJDK project[1]?  It seems to me this "increasing buffer"
> behaviour would be useful more generally...

Done.

Joerg

[Jeroen Reijn-3]

If I want to upload the jars into the maven repository (i already did
for 2.1.11) I also have to add my public key to the KEYS file as well.

Now I'm a bit confused. I noticed there is a KEYS file in the svn [1]
for the 2_1_X branch, but there is also one on [2].

Which one do I need to update with my public key and how do I do that?
Can I just run "pgp -kxa <your name>" as stated at [3] or do I need to
do something special?

I guess I'm a bit of a newbie with the PGP stuff. All help will be
appreciated.

[1]https://svn.apache.org/repos/asf/cocoon/branches/BRANCH_2_1_X/
[2]http://www.apache.org/dist/cocoon/KEYS
[3]http://www.apache.org/dev/release-signing.html#keys-policy

Carsten Ziegeler wrote:

> Joerg Heinicke wrote:
>> On 28.05.2008 02:05, Carsten Ziegeler wrote:
>>
>> If it is only 2.1 (at least for the time being) I can do this job
>> (though I'm neither working with Cocoon anymore lately). Just have to
>> check this documentation (any links?) what it needs to do such a
>> release (signed PGP key or things like that).
> Here is the whole "guide" :)
> http://wiki.apache.org/cocoon/CocoonReleaseHowTo
>
> You need a key which you have to put up on our site into the KEYS file.
>
> Now, I can do the release - although I think it makes more sense someone
> more close to the project than I am should do the work. Whatever you
> prefer.
>
>> Only issue I want to solve before the release is the
>> BufferedOutputStream issue. I planned to do it this weekend.
> Great
>
> Carsten
>

[Jeroen Reijn-3]

Hmm I was not finished typing my email (stupid mail client), but want I
wanted to say is that since Joerg will also come across this issue I
guess, maybe somebody can help us out?

Jeroen Reijn wrote:

> If I want to upload the jars into the maven repository (i already did
> for 2.1.11) I also have to add my public key to the KEYS file as well.
>
> Now I'm a bit confused. I noticed there is a KEYS file in the svn [1]
> for the 2_1_X branch, but there is also one on [2].
>
> Which one do I need to update with my public key and how do I do that?
> Can I just run "pgp -kxa <your name>" as stated at [3] or do I need to
> do something special?
>
> I guess I'm a bit of a newbie with the PGP stuff. All help will be
> appreciated.
>
> [1]https://svn.apache.org/repos/asf/cocoon/branches/BRANCH_2_1_X/
> [2]http://www.apache.org/dist/cocoon/KEYS
> [3]http://www.apache.org/dev/release-signing.html#keys-policy
>
> Carsten Ziegeler wrote:
>> Joerg Heinicke wrote:
>>> On 28.05.2008 02:05, Carsten Ziegeler wrote:
>>>
>>> If it is only 2.1 (at least for the time being) I can do this job
>>> (though I'm neither working with Cocoon anymore lately). Just have to
>>> check this documentation (any links?) what it needs to do such a
>>> release (signed PGP key or things like that).
>> Here is the whole "guide" :)
>> http://wiki.apache.org/cocoon/CocoonReleaseHowTo
>>
>> You need a key which you have to put up on our site into the KEYS file.
>>
>> Now, I can do the release - although I think it makes more sense
>> someone more close to the project than I am should do the work.
>> Whatever you prefer.
>>
>>> Only issue I want to solve before the release is the
>>> BufferedOutputStream issue. I planned to do it this weekend.
>> Great
>>
>> Carsten
>>

[Carsten Ziegeler]

Where are we now with the release?

Carsten

--
Carsten Ziegeler
cziegeler@...

[Andrew Savory]

Hi,

2008/7/3 Carsten Ziegeler <cziegeler@...>:

Where are we now with the release?

I have some fixes to the fix I committed last week, but won't have a chance to look at it again until next week earliest ... I can roll back if others want to push the release.

Andrew.
--
asavory@... / contact@...
http://www.andrewsavory.com/