You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cocoon.apache.org by Joerg Heinicke <jo...@gmx.de> on 2008/06/02 06:42:23 UTC
BufferedOutputStream (was: Releasing 2.1.12 - a new release manager?)
On 29.05.2008 00:06, Joerg Heinicke wrote:
> Only issue I want to solve before the release is the
> BufferedOutputStream issue. I planned to do it this weekend.
Done. Please review the file attached. It's still completely untested.
At the moment I need some sleep ;) I will write junit tests for it this
week and eventually commit it.
Joerg
Re: BufferedOutputStream
Posted by Sylvain Wallez <sy...@apache.org>.
Joerg Heinicke wrote:
> On 02.06.2008 05:56, Sylvain Wallez wrote:
>
>>>> Only issue I want to solve before the release is the
>>>> BufferedOutputStream issue. I planned to do it this weekend.
>>>
>>> Done. Please review the file attached. It's still completely
>>> untested. At the moment I need some sleep ;) I will write junit
>>> tests for it this week and eventually commit it.
>>
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This
> is also possible with java.io.BufferedOutputStream, if the buffer size
> is big enough (current default value is 1MB), but then the buffer
> byte[] is always that big rather than increasing. That's what's
> happening right now, if you don't specify -1 as buffer size. -1 means
> complete buffering which on the other hand might lead to
> OutOfMemoryError [1]. In addition our BOS counts the bytes so that we
> can use the value to set the content length header.
Got it. I'm using a similar technique with a servlet filter providing
the response with a ByteArrayOutputStream that gives the size and has
the convenient writeTo(OutputStream) method. Now in my case [1] pages
are always small and I don't need to put a high limit on the size of the
buffer.
Sylvain
[1] http://bluxte.net/blog/2008-04/29-54-29.html
--
Sylvain Wallez - http://bluxte.net
Re: BufferedOutputStream
Posted by Joerg Heinicke <jo...@gmx.de>.
On 07.06.2008 03:13, Andy Stevens wrote:
> Out of interest, have you raised an enhancement request and/or a patch
> with the OpenJDK project[1]? It seems to me this "increasing buffer"
> behaviour would be useful more generally...
Done.
Joerg
Re: BufferedOutputStream
Posted by Andy Stevens <in...@googlemail.com>.
Joerg Heinicke wrote:
> On 02.06.2008 05:56, Sylvain Wallez wrote:
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This is
> also possible with java.io.BufferedOutputStream, if the buffer size is
> big enough (current default value is 1MB), but then the buffer byte[] is
> always that big rather than increasing. That's what's happening right
> now, if you don't specify -1 as buffer size. -1 means complete buffering
> which on the other hand might lead to OutOfMemoryError [1]. In addition
> our BOS counts the bytes so that we can use the value to set the content
> length header.
>
> Joerg
>
> [1] https://issues.apache.org/jira/browse/COCOON-2168
Out of interest, have you raised an enhancement request and/or a patch
with the OpenJDK project[1]? It seems to me this "increasing buffer"
behaviour would be useful more generally...
Andy.
[1] http://openjdk.java.net/
--
http://pseudoq.sourceforge.net/ Open source Sudoku solver
Re: BufferedOutputStream
Posted by Joerg Heinicke <jo...@gmx.de>.
Reinhard Pötz <reinhard <at> apache.org> writes:
> >> Is my understanding right that the content length header can only be set
> >> as long as you haven't written into the underlying output stream?
> >
> > Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
> > have been flushed yet.
>
> What happens when the buffer size is exceeded? Does it mean that the
> output is streamed and the content length parameter can't be set?
If the buffer size is exceeded the output stream is flushed and eventually
written to the response. A header like the content-length can no longer be set.
That's why Cocoon's BufferedOutputStream was setup to buffer all and everything
by default which lead to COCOON-2168 [1] with Cocoon's default setting. With the
change to a buffer of 1 MB rather than endless buffering we switched back to
java.io.BufferedOutputStream, but this allocates the memory for the buffer
immediately. So even with 10 KB pages you always end up with 1 MB byte[]. I
intend to fix this in our BufferedOutputStream with an increasing buffer and the
two configuration parameters initial buffer size and flush buffer size.
Just to make it clear: We talk about default configuration of Cocoon here. You
still can enforce complete buffering by setting flush buffer size to -1 again.
But it's up to the user to enforce this and potentially run into OOME. It's also
up to the users to find a reasonable flush buffer size as it always used to be
when they haven't used the endless buffering. Only thing that is going to change
is in case of not endless buffering: The full buffer is not allocated
immediately but step by step.
Joerg
[1] https://issues.apache.org/jira/browse/COCOON-2168
Re: BufferedOutputStream
Posted by Reinhard Pötz <re...@apache.org>.
Joerg Heinicke wrote:
> Reinhard Pötz <reinhard <at> apache.org> writes:
>
>> Is my understanding right that the content length header can only be set
>> as long as you haven't written into the underlying output stream?
>
> Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
> have been flushed yet.
What happens when the buffer size is exceeded? Does it mean that the
output is streamed and the content length parameter can't be set?
--
Reinhard Pötz Managing Director, {Indoqa} GmbH
http://www.indoqa.com/en/people/reinhard.poetz/
Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member reinhard@apache.org
________________________________________________________________________
Re: BufferedOutputStream
Posted by Joerg Heinicke <jo...@gmx.de>.
Reinhard Pötz <reinhard <at> apache.org> writes:
> Is my understanding right that the content length header can only be set
> as long as you haven't written into the underlying output stream?
Yes, it is. That's why the buffering has to be big enough, i.e. nothing must
have been flushed yet.
Joerg
Re: BufferedOutputStream
Posted by Reinhard Pötz <re...@apache.org>.
Joerg Heinicke wrote:
> On 02.06.2008 05:56, Sylvain Wallez wrote:
>
>>>> Only issue I want to solve before the release is the
>>>> BufferedOutputStream issue. I planned to do it this weekend.
>>>
>>> Done. Please review the file attached. It's still completely
>>> untested. At the moment I need some sleep ;) I will write junit tests
>>> for it this week and eventually commit it.
>>
>> Stupid question: why do we need a special BufferedOutputStream?
>
> For being able to reset the response buffer for error handling. This is
> also possible with java.io.BufferedOutputStream, if the buffer size is
> big enough (current default value is 1MB), but then the buffer byte[] is
> always that big rather than increasing. That's what's happening right
> now, if you don't specify -1 as buffer size. -1 means complete buffering
> which on the other hand might lead to OutOfMemoryError [1]. In addition
> our BOS counts the bytes so that we can use the value to set the content
> length header.
Is my understanding right that the content length header can only be set
as long as you haven't written into the underlying output stream?
--
Reinhard Pötz Managing Director, {Indoqa} GmbH
http://www.indoqa.com/en/people/reinhard.poetz/
Member of the Apache Software Foundation
Apache Cocoon Committer, PMC member reinhard@apache.org
________________________________________________________________________
Re: BufferedOutputStream
Posted by Joerg Heinicke <jo...@gmx.de>.
On 02.06.2008 05:56, Sylvain Wallez wrote:
>>> Only issue I want to solve before the release is the
>>> BufferedOutputStream issue. I planned to do it this weekend.
>>
>> Done. Please review the file attached. It's still completely untested.
>> At the moment I need some sleep ;) I will write junit tests for it
>> this week and eventually commit it.
>
> Stupid question: why do we need a special BufferedOutputStream?
For being able to reset the response buffer for error handling. This is
also possible with java.io.BufferedOutputStream, if the buffer size is
big enough (current default value is 1MB), but then the buffer byte[] is
always that big rather than increasing. That's what's happening right
now, if you don't specify -1 as buffer size. -1 means complete buffering
which on the other hand might lead to OutOfMemoryError [1]. In addition
our BOS counts the bytes so that we can use the value to set the content
length header.
Joerg
[1] https://issues.apache.org/jira/browse/COCOON-2168
Re: BufferedOutputStream
Posted by Sylvain Wallez <sy...@apache.org>.
Joerg Heinicke wrote:
> On 29.05.2008 00:06, Joerg Heinicke wrote:
>
>> Only issue I want to solve before the release is the
>> BufferedOutputStream issue. I planned to do it this weekend.
>
> Done. Please review the file attached. It's still completely untested.
> At the moment I need some sleep ;) I will write junit tests for it
> this week and eventually commit it.
Stupid question: why do we need a special BufferedOutputStream?
Sylvain
--
Sylvain Wallez - http://bluxte.net