You are viewing a plain text version of this content. The canonical link for it is here.
Posted to httpclient-users@hc.apache.org by sreenivas somavarapu <so...@gmail.com> on 2023/03/06 10:56:39 UTC

AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Hi Team,

We are using AbstractClassicEntityProducer with a customization of storing
/ computing contentLength as well which works fine if we have small POST
data, but when POST data is big (for example 64 KB), it seems OutPutstream
(internally using ContentOutputStream a wrapper over SharedOutputBuffer of
initial size 1024) is not able to write data (We are using temporary buffer
block of 4kb but still it is not able to write 4KB of data). Is there any
limitation or some configuration problem?

*Code being used*
        SharedOutputBuffer buffer = new SharedOutputBuffer(1024);

        produceData(contentType, new ContentOutputStream(buffer));

protected void produceData(ContentType contentType, OutputStream
outputStream) throws IOException {
        byte[] tmp = new byte[4 * 1024];
        int l = -1;
        while ((l = inputStream.read(tmp)) != -1) {
*            outputStream.write(tmp, 0, l);  <---- This is where request
hangs / waits*
        }
}

*HTTPClient has last line as below*
2023-03-02 21:53:52,710 DEBUG
[org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec]
ex-0000000004: produce request data

*StackTrace when we shutdown connection manager*
 java.io.InterruptedIOException
at
org.apache.hc.core5.http.nio.support.classic.SharedOutputBuffer.waitFlush(SharedOutputBuffer.java:160)
at
org.apache.hc.core5.http.nio.support.classic.SharedOutputBuffer.write(SharedOutputBuffer.java:95)
at
org.apache.hc.core5.http.nio.support.classic.ContentOutputStream.write(ContentOutputStream.java:61)
at com.apache.httpclient.entity.AsyncRequestEntity.produceData(Unknown
Source)
at com.apache.httpclient.entity.AsyncRequestEntity.produce(Unknown Source)
at
org.apache.hc.core5.http.nio.support.BasicRequestProducer.produce(BasicRequestProducer.java:104)
at
org.apache.hc.client5.http.impl.async.InternalAbstractHttpAsyncClient$2.produce(InternalAbstractHttpAsyncClient.java:260)
at
org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.produce(HttpAsyncMainClientExec.java:163)
at
org.apache.hc.client5.http.impl.async.LoggingAsyncClientExchangeHandler.produce(LoggingAsyncClientExchangeHandler.java:90)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.commitRequest(ClientHttp1StreamHandler.java:171)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.lambda$produceOutput$0(ClientHttp1StreamHandler.java:183)
at
org.apache.hc.client5.http.impl.async.LoggingAsyncClientExchangeHandler.lambda$produceRequest$0(LoggingAsyncClientExchangeHandler.java:76)
at
org.apache.hc.client5.http.impl.async.HttpAsyncMainClientExec$1.produceRequest(HttpAsyncMainClientExec.java:150)
at
org.apache.hc.client5.http.impl.async.LoggingAsyncClientExchangeHandler.produceRequest(LoggingAsyncClientExchangeHandler.java:71)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamHandler.produceOutput(ClientHttp1StreamHandler.java:183)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1StreamDuplexer.execute(ClientHttp1StreamDuplexer.java:326)
at
org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.processCommands(AbstractHttp1StreamDuplexer.java:239)
at
org.apache.hc.core5.http.impl.nio.AbstractHttp1StreamDuplexer.onOutput(AbstractHttp1StreamDuplexer.java:386)
at
org.apache.hc.core5.http.impl.nio.AbstractHttp1IOEventHandler.outputReady(AbstractHttp1IOEventHandler.java:73)
at
org.apache.hc.core5.http.impl.nio.ClientHttp1IOEventHandler.outputReady(ClientHttp1IOEventHandler.java:41)
at
org.apache.hc.client5.http.impl.async.LoggingIOSession$1.outputReady(LoggingIOSession.java:243)
at
org.apache.hc.core5.reactor.ssl.SSLIOSession.encryptData(SSLIOSession.java:639)
at
org.apache.hc.core5.reactor.ssl.SSLIOSession.access$400(SSLIOSession.java:74)
at
org.apache.hc.core5.reactor.ssl.SSLIOSession$1.outputReady(SSLIOSession.java:208)
at
org.apache.hc.core5.reactor.InternalDataChannel.onIOEvent(InternalDataChannel.java:152)
at
org.apache.hc.core5.reactor.InternalChannel.handleIOEvent(InternalChannel.java:51)
at
org.apache.hc.core5.reactor.SingleCoreIOReactor.processEvents(SingleCoreIOReactor.java:178)
at
org.apache.hc.core5.reactor.SingleCoreIOReactor.doExecute(SingleCoreIOReactor.java:127)
at
org.apache.hc.core5.reactor.AbstractSingleCoreIOReactor.execute(AbstractSingleCoreIOReactor.java:86)
at org.apache.hc.core5.reactor.IOReactorWorker.run(IOReactorWorker.java:44)
at java.lang.Thread.run(Thread.java:748)

*Another observation is after connecting to a server, the request is not
getting timed out even after 30 minutes as well in the above scenario
(i.e., Large POST Data). DefaultRequestConfig ResponseTimeout is being set
to 5 seconds and even DefaultConnectionConfig ConnectionTimeout and
SocketTimeout also being set to 180000 milliseconds (ie., 3 minutes) and
also same value is being set to IOReactorConfig SoTimeout as well.  Is
there any other setting which is missing which is causing this behavior?*

*HTTPClient log extract*
2023-03-02 21:31:51,187 DEBUG
[org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager]
ep-0000000001 connecting endpoint to https://server.us.com (180000
MILLISECONDS)

2023-03-02 21:31:55,964 DEBUG
[org.apache.hc.client5.http.impl.nio.PoolingAsyncClientConnectionManager]
ep-0000000001 connection c-0000000000 can be kept alive for 3 MINUTES

--
Cheers,
S. Sreenivas

Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by sreenivas somavarapu <so...@gmail.com>.
Hi Oleg,

*public final class SharedOutputBuffer extends ExpandableBuffer implements
ContentOutputBuffer*

Above method definition (*extends ExpandableBuffer*) made me think that
SharedOutputBuffer is expandable.

Regards,
Sreenivas

On Thu, Mar 9, 2023 at 8:20 PM sreenivas somavarapu <
soma.sreenivas@gmail.com> wrote:

> Hi Oleg,
>
> *if (src.remaining() < 1024 && buffer().remaining() > src.remaining()) {*
>
> Problem with this above condition is that even though buffer has capacity
> to store complete contents of src, this condition will not satisfy if src
> has 1024 bytes or more data (may well be in limit of actual size provided
> during creation of buffer) to be written to buffer
>
> Simple way to reproduce the problem is (Same has been demonstrated with
> sample program in previous mail)
> -    Create SharedOutputBuffer with 4096 bytes of capacity
> -    Now try to write 1024 bytes of data to above created buffer
>
> second step still will cause in a hang as first if condition is not
> satisfied due to *src.remaining() < 1024* even though there is enough
> capacity to store data and it proceeds to second condition which will cause
> the program to wait on a condition for flush (even though this is the first
> write we are performing)
>
>
> *if (buffer().position() > 0 || dataStreamChannel == null) {*
>
> *                         waitFlush();*
> *}*
>
> Regards,
> Sreenivas
>
> On Mon, Mar 6, 2023 at 9:12 PM Oleg Kalnichevski <ol...@apache.org> wrote:
>
>> On Mon, 2023-03-06 at 21:03 +0530, sreenivas somavarapu wrote:
>> > Hi Oleg,
>> >
>> > My sample program (it's a kind of unit test for SharedOutputBuffer to
>> > demonstrate that it is not able to write single byte if input byte
>> > array
>> > size is greater than 1023) doesn't have flush as my intention is to
>> > show
>> > the issue with SharedOutputBuffer alone and it is not a complete
>> > program
>> > which sends the data to the actual server and get response back. Main
>> > issue
>> > lies while writing data greater than 1023 bytes at first go itself
>> > to SharedOutputBuffer from the byte array.
>> >
>> > I think the below conditions in SharedOutputBuffer is causing this
>> > behavior
>> >
>> >             while (src.hasRemaining()) {
>> >                 // always buffer small chunks
>> >                 *if (src.remaining() < 1024 && buffer().remaining() >
>> > src.remaining()) {*
>> >                     buffer().put(src);
>> >                 } else {
>> >                    * if (buffer().position() > 0 || dataStreamChannel
>> > ==
>> > null) {*
>> >                         waitFlush();
>> >                     }
>> >
>>
>> What behavior? This code blocks if it is unable to store input data
>> until the internal buffer gets flushed and space in it frees up, which
>> is what is it supposed to do.
>>
>> If you think SharedOutputBuffer is wrong, please just use your own
>> custom buffer implementation.
>>
>> Oleg
>>
>>
>> > Regards,
>> > Sreenivas
>> >
>> > On Mon, Mar 6, 2023 at 8:52 PM Oleg Kalnichevski <ol...@apache.org>
>> > wrote:
>> >
>> > > On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
>> > > > Hi Oleg,
>> > > >
>> > > > It looks like some issue with SharedOutputBuffer implementation
>> > > > and
>> > > > it is
>> > > > also not expandable as we have to allocate complete size during
>> > > > initialization itself. If I change below two lines from my
>> > > > program it
>> > > > proceeds further and completes the request (Basically
>> > > > SharedOutputBuffer is
>> > > > not able to write 1024 or above bytes of data at single go (If we
>> > > > declare
>> > > > temp block size as 1023 also it works)). It looks like there is
>> > > > an
>> > > > explicit 1024 length check in SharedOutputBuffer which is causing
>> > > > this
>> > > > behavior.
>> > > >
>> > > > SharedOutputBuffer buffer = new
>> > > > SharedOutputBuffer(req.contentLength() +
>> > > > 1024);
>> > >
>> > > What you are doing is _completely_ wrong. SharedOutputBuffer is not
>> > > meant to be expandable.
>> > >
>> > > You cannot just write into the buffer and expect it to work by
>> > > magic.
>> > > You also must flush the buffer.
>> > >
>> > > Oleg
>> > >
>> > > >
>> > > > byte[] tmp = new byte[512];
>> > > >
>> > > > Earlier those 2 lines were as below
>> > > >
>> > > >  SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
>> > > >
>> > > > byte[] tmp = new byte[4 * 1024];
>> > > >
>> > > > Below is my sample program on SharedOutputBuffer  which
>> > > > demonstrated
>> > > > this
>> > > > behavior
>> > > >
>> > > >         final StringBuilder sbuffer = new StringBuilder("123");
>> > > >         System.out.println("O/P: " + sbuffer.toString());
>> > > >         for (long i = 0; i < (64 * 1000); ++i) {
>> > > >             sbuffer.append('A');
>> > > >         }
>> > > >
>> > > >         String postData = sbuffer.toString();
>> > > >         System.out.println("O/P1: " + postData);
>> > > >
>> > > >         byte[] inputData =
>> > > > postData.getBytes(StandardCharsets.UTF_8);
>> > > >         ByteArrayInputStream inputStream = new
>> > > > ByteArrayInputStream(inputData);
>> > > >         SharedOutputBuffer outputStream = new
>> > > > SharedOutputBuffer(100
>> > > > *
>> > > > 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
>> > > >         *byte[] tmp = new byte[1024];  <-- If we change this to
>> > > > 1023
>> > > > then
>> > > > it works*
>> > > >         int l = -1;
>> > > >         while ((l = inputStream.read(tmp)) != -1) {
>> > > >             System.out.println("Writing Data");
>> > > >             System.out.println(l);
>> > > >             outputStream.write(tmp, 0, l);
>> > > >             System.out.println("Wrote Data");
>> > > >         }
>> > > >
>> > > >         System.out.println("Done Writing Data");
>> > > >         outputStream.writeCompleted();
>> > > >         System.out.println(outputStream.toString());
>> > > >
>> > > > Regards,
>> > > > Sreenivas
>> > > >
>> > > > On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski
>> > > > <ol...@apache.org>
>> > > > wrote:
>> > > >
>> > > > > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
>> > > > > > Hi Team,
>> > > > > >
>> > > > > > We are using AbstractClassicEntityProducer with a
>> > > > > > customization
>> > > > > > of
>> > > > > > storing
>> > > > > > / computing contentLength as well which works fine if we have
>> > > > > > small
>> > > > > > POST
>> > > > > > data, but when POST data is big (for example 64 KB), it seems
>> > > > > > OutPutstream
>> > > > > > (internally using ContentOutputStream a wrapper over
>> > > > > > SharedOutputBuffer of
>> > > > > > initial size 1024) is not able to write data (We are using
>> > > > > > temporary
>> > > > > > buffer
>> > > > > > block of 4kb but still it is not able to write 4KB of data).
>> > > > > > Is
>> > > > > > there
>> > > > > > any
>> > > > > > limitation or some configuration problem?
>> > > > > >
>> > > > > > *Code being used*
>> > > > > >         SharedOutputBuffer buffer = new
>> > > > > > SharedOutputBuffer(1024);
>> > > > > >
>> > > > > >         produceData(contentType, new
>> > > > > > ContentOutputStream(buffer));
>> > > > > >
>> > > > > > protected void produceData(ContentType contentType,
>> > > > > > OutputStream
>> > > > > > outputStream) throws IOException {
>> > > > > >         byte[] tmp = new byte[4 * 1024];
>> > > > > >         int l = -1;
>> > > > > >         while ((l = inputStream.read(tmp)) != -1) {
>> > > > > > *            outputStream.write(tmp, 0, l);  <---- This is
>> > > > > > where
>> > > > > > request
>> > > > > > hangs / waits*
>> > > > >
>> > > > > This is likely due to a bug in your code that prevents the
>> > > > > shared
>> > > > > buffer from being flushed.
>> > > > >
>> > > > > Oleg
>> > > > >
>> > > > >
>> > > > > ---------------------------------------------------------------
>> > > > > ----
>> > > > > --
>> > > > > To unsubscribe, e-mail:
>> > > > > httpclient-users-unsubscribe@hc.apache.org
>> > > > > For additional commands, e-mail:
>> > > > > httpclient-users-help@hc.apache.org
>> > > > >
>> > > > >
>> > > >
>> > >
>> > >
>> > > -------------------------------------------------------------------
>> > > --
>> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> > > For additional commands, e-mail:
>> > > httpclient-users-help@hc.apache.org
>> > >
>> > >
>> >
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
>> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>>
>>
>
> --
> Cheers,
> S. Sreenivas
>


-- 
Cheers,
S. Sreenivas

Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by sreenivas somavarapu <so...@gmail.com>.
Hi Oleg,

*if (src.remaining() < 1024 && buffer().remaining() > src.remaining()) {*

Problem with this above condition is that even though buffer has capacity
to store complete contents of src, this condition will not satisfy if src
has 1024 bytes or more data (may well be in limit of actual size provided
during creation of buffer) to be written to buffer

Simple way to reproduce the problem is (Same has been demonstrated with
sample program in previous mail)
-    Create SharedOutputBuffer with 4096 bytes of capacity
-    Now try to write 1024 bytes of data to above created buffer

second step still will cause in a hang as first if condition is not
satisfied due to *src.remaining() < 1024* even though there is enough
capacity to store data and it proceeds to second condition which will cause
the program to wait on a condition for flush (even though this is the first
write we are performing)


*if (buffer().position() > 0 || dataStreamChannel == null) {*

*                         waitFlush();*
*}*

Regards,
Sreenivas

On Mon, Mar 6, 2023 at 9:12 PM Oleg Kalnichevski <ol...@apache.org> wrote:

> On Mon, 2023-03-06 at 21:03 +0530, sreenivas somavarapu wrote:
> > Hi Oleg,
> >
> > My sample program (it's a kind of unit test for SharedOutputBuffer to
> > demonstrate that it is not able to write single byte if input byte
> > array
> > size is greater than 1023) doesn't have flush as my intention is to
> > show
> > the issue with SharedOutputBuffer alone and it is not a complete
> > program
> > which sends the data to the actual server and get response back. Main
> > issue
> > lies while writing data greater than 1023 bytes at first go itself
> > to SharedOutputBuffer from the byte array.
> >
> > I think the below conditions in SharedOutputBuffer is causing this
> > behavior
> >
> >             while (src.hasRemaining()) {
> >                 // always buffer small chunks
> >                 *if (src.remaining() < 1024 && buffer().remaining() >
> > src.remaining()) {*
> >                     buffer().put(src);
> >                 } else {
> >                    * if (buffer().position() > 0 || dataStreamChannel
> > ==
> > null) {*
> >                         waitFlush();
> >                     }
> >
>
> What behavior? This code blocks if it is unable to store input data
> until the internal buffer gets flushed and space in it frees up, which
> is what is it supposed to do.
>
> If you think SharedOutputBuffer is wrong, please just use your own
> custom buffer implementation.
>
> Oleg
>
>
> > Regards,
> > Sreenivas
> >
> > On Mon, Mar 6, 2023 at 8:52 PM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> >
> > > On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
> > > > Hi Oleg,
> > > >
> > > > It looks like some issue with SharedOutputBuffer implementation
> > > > and
> > > > it is
> > > > also not expandable as we have to allocate complete size during
> > > > initialization itself. If I change below two lines from my
> > > > program it
> > > > proceeds further and completes the request (Basically
> > > > SharedOutputBuffer is
> > > > not able to write 1024 or above bytes of data at single go (If we
> > > > declare
> > > > temp block size as 1023 also it works)). It looks like there is
> > > > an
> > > > explicit 1024 length check in SharedOutputBuffer which is causing
> > > > this
> > > > behavior.
> > > >
> > > > SharedOutputBuffer buffer = new
> > > > SharedOutputBuffer(req.contentLength() +
> > > > 1024);
> > >
> > > What you are doing is _completely_ wrong. SharedOutputBuffer is not
> > > meant to be expandable.
> > >
> > > You cannot just write into the buffer and expect it to work by
> > > magic.
> > > You also must flush the buffer.
> > >
> > > Oleg
> > >
> > > >
> > > > byte[] tmp = new byte[512];
> > > >
> > > > Earlier those 2 lines were as below
> > > >
> > > >  SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> > > >
> > > > byte[] tmp = new byte[4 * 1024];
> > > >
> > > > Below is my sample program on SharedOutputBuffer  which
> > > > demonstrated
> > > > this
> > > > behavior
> > > >
> > > >         final StringBuilder sbuffer = new StringBuilder("123");
> > > >         System.out.println("O/P: " + sbuffer.toString());
> > > >         for (long i = 0; i < (64 * 1000); ++i) {
> > > >             sbuffer.append('A');
> > > >         }
> > > >
> > > >         String postData = sbuffer.toString();
> > > >         System.out.println("O/P1: " + postData);
> > > >
> > > >         byte[] inputData =
> > > > postData.getBytes(StandardCharsets.UTF_8);
> > > >         ByteArrayInputStream inputStream = new
> > > > ByteArrayInputStream(inputData);
> > > >         SharedOutputBuffer outputStream = new
> > > > SharedOutputBuffer(100
> > > > *
> > > > 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
> > > >         *byte[] tmp = new byte[1024];  <-- If we change this to
> > > > 1023
> > > > then
> > > > it works*
> > > >         int l = -1;
> > > >         while ((l = inputStream.read(tmp)) != -1) {
> > > >             System.out.println("Writing Data");
> > > >             System.out.println(l);
> > > >             outputStream.write(tmp, 0, l);
> > > >             System.out.println("Wrote Data");
> > > >         }
> > > >
> > > >         System.out.println("Done Writing Data");
> > > >         outputStream.writeCompleted();
> > > >         System.out.println(outputStream.toString());
> > > >
> > > > Regards,
> > > > Sreenivas
> > > >
> > > > On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski
> > > > <ol...@apache.org>
> > > > wrote:
> > > >
> > > > > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > > > > > Hi Team,
> > > > > >
> > > > > > We are using AbstractClassicEntityProducer with a
> > > > > > customization
> > > > > > of
> > > > > > storing
> > > > > > / computing contentLength as well which works fine if we have
> > > > > > small
> > > > > > POST
> > > > > > data, but when POST data is big (for example 64 KB), it seems
> > > > > > OutPutstream
> > > > > > (internally using ContentOutputStream a wrapper over
> > > > > > SharedOutputBuffer of
> > > > > > initial size 1024) is not able to write data (We are using
> > > > > > temporary
> > > > > > buffer
> > > > > > block of 4kb but still it is not able to write 4KB of data).
> > > > > > Is
> > > > > > there
> > > > > > any
> > > > > > limitation or some configuration problem?
> > > > > >
> > > > > > *Code being used*
> > > > > >         SharedOutputBuffer buffer = new
> > > > > > SharedOutputBuffer(1024);
> > > > > >
> > > > > >         produceData(contentType, new
> > > > > > ContentOutputStream(buffer));
> > > > > >
> > > > > > protected void produceData(ContentType contentType,
> > > > > > OutputStream
> > > > > > outputStream) throws IOException {
> > > > > >         byte[] tmp = new byte[4 * 1024];
> > > > > >         int l = -1;
> > > > > >         while ((l = inputStream.read(tmp)) != -1) {
> > > > > > *            outputStream.write(tmp, 0, l);  <---- This is
> > > > > > where
> > > > > > request
> > > > > > hangs / waits*
> > > > >
> > > > > This is likely due to a bug in your code that prevents the
> > > > > shared
> > > > > buffer from being flushed.
> > > > >
> > > > > Oleg
> > > > >
> > > > >
> > > > > ---------------------------------------------------------------
> > > > > ----
> > > > > --
> > > > > To unsubscribe, e-mail:
> > > > > httpclient-users-unsubscribe@hc.apache.org
> > > > > For additional commands, e-mail:
> > > > > httpclient-users-help@hc.apache.org
> > > > >
> > > > >
> > > >
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

-- 
Cheers,
S. Sreenivas

Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2023-03-06 at 21:03 +0530, sreenivas somavarapu wrote:
> Hi Oleg,
> 
> My sample program (it's a kind of unit test for SharedOutputBuffer to
> demonstrate that it is not able to write single byte if input byte
> array
> size is greater than 1023) doesn't have flush as my intention is to
> show
> the issue with SharedOutputBuffer alone and it is not a complete
> program
> which sends the data to the actual server and get response back. Main
> issue
> lies while writing data greater than 1023 bytes at first go itself
> to SharedOutputBuffer from the byte array.
> 
> I think the below conditions in SharedOutputBuffer is causing this
> behavior
>
>             while (src.hasRemaining()) {
>                 // always buffer small chunks
>                 *if (src.remaining() < 1024 && buffer().remaining() >
> src.remaining()) {*
>                     buffer().put(src);
>                 } else {
>                    * if (buffer().position() > 0 || dataStreamChannel
> ==
> null) {*
>                         waitFlush();
>                     }
> 

What behavior? This code blocks if it is unable to store input data
until the internal buffer gets flushed and space in it frees up, which
is what is it supposed to do.

If you think SharedOutputBuffer is wrong, please just use your own
custom buffer implementation.

Oleg


> Regards,
> Sreenivas
> 
> On Mon, Mar 6, 2023 at 8:52 PM Oleg Kalnichevski <ol...@apache.org>
> wrote:
> 
> > On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
> > > Hi Oleg,
> > > 
> > > It looks like some issue with SharedOutputBuffer implementation
> > > and
> > > it is
> > > also not expandable as we have to allocate complete size during
> > > initialization itself. If I change below two lines from my
> > > program it
> > > proceeds further and completes the request (Basically
> > > SharedOutputBuffer is
> > > not able to write 1024 or above bytes of data at single go (If we
> > > declare
> > > temp block size as 1023 also it works)). It looks like there is
> > > an
> > > explicit 1024 length check in SharedOutputBuffer which is causing
> > > this
> > > behavior.
> > > 
> > > SharedOutputBuffer buffer = new
> > > SharedOutputBuffer(req.contentLength() +
> > > 1024);
> > 
> > What you are doing is _completely_ wrong. SharedOutputBuffer is not
> > meant to be expandable.
> > 
> > You cannot just write into the buffer and expect it to work by
> > magic.
> > You also must flush the buffer.
> > 
> > Oleg
> > 
> > > 
> > > byte[] tmp = new byte[512];
> > > 
> > > Earlier those 2 lines were as below
> > > 
> > >  SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> > > 
> > > byte[] tmp = new byte[4 * 1024];
> > > 
> > > Below is my sample program on SharedOutputBuffer  which
> > > demonstrated
> > > this
> > > behavior
> > > 
> > >         final StringBuilder sbuffer = new StringBuilder("123");
> > >         System.out.println("O/P: " + sbuffer.toString());
> > >         for (long i = 0; i < (64 * 1000); ++i) {
> > >             sbuffer.append('A');
> > >         }
> > > 
> > >         String postData = sbuffer.toString();
> > >         System.out.println("O/P1: " + postData);
> > > 
> > >         byte[] inputData =
> > > postData.getBytes(StandardCharsets.UTF_8);
> > >         ByteArrayInputStream inputStream = new
> > > ByteArrayInputStream(inputData);
> > >         SharedOutputBuffer outputStream = new
> > > SharedOutputBuffer(100
> > > *
> > > 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
> > >         *byte[] tmp = new byte[1024];  <-- If we change this to
> > > 1023
> > > then
> > > it works*
> > >         int l = -1;
> > >         while ((l = inputStream.read(tmp)) != -1) {
> > >             System.out.println("Writing Data");
> > >             System.out.println(l);
> > >             outputStream.write(tmp, 0, l);
> > >             System.out.println("Wrote Data");
> > >         }
> > > 
> > >         System.out.println("Done Writing Data");
> > >         outputStream.writeCompleted();
> > >         System.out.println(outputStream.toString());
> > > 
> > > Regards,
> > > Sreenivas
> > > 
> > > On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski
> > > <ol...@apache.org>
> > > wrote:
> > > 
> > > > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > > > > Hi Team,
> > > > > 
> > > > > We are using AbstractClassicEntityProducer with a
> > > > > customization
> > > > > of
> > > > > storing
> > > > > / computing contentLength as well which works fine if we have
> > > > > small
> > > > > POST
> > > > > data, but when POST data is big (for example 64 KB), it seems
> > > > > OutPutstream
> > > > > (internally using ContentOutputStream a wrapper over
> > > > > SharedOutputBuffer of
> > > > > initial size 1024) is not able to write data (We are using
> > > > > temporary
> > > > > buffer
> > > > > block of 4kb but still it is not able to write 4KB of data).
> > > > > Is
> > > > > there
> > > > > any
> > > > > limitation or some configuration problem?
> > > > > 
> > > > > *Code being used*
> > > > >         SharedOutputBuffer buffer = new
> > > > > SharedOutputBuffer(1024);
> > > > > 
> > > > >         produceData(contentType, new
> > > > > ContentOutputStream(buffer));
> > > > > 
> > > > > protected void produceData(ContentType contentType,
> > > > > OutputStream
> > > > > outputStream) throws IOException {
> > > > >         byte[] tmp = new byte[4 * 1024];
> > > > >         int l = -1;
> > > > >         while ((l = inputStream.read(tmp)) != -1) {
> > > > > *            outputStream.write(tmp, 0, l);  <---- This is
> > > > > where
> > > > > request
> > > > > hangs / waits*
> > > > 
> > > > This is likely due to a bug in your code that prevents the
> > > > shared
> > > > buffer from being flushed.
> > > > 
> > > > Oleg
> > > > 
> > > > 
> > > > ---------------------------------------------------------------
> > > > ----
> > > > --
> > > > To unsubscribe, e-mail:
> > > > httpclient-users-unsubscribe@hc.apache.org
> > > > For additional commands, e-mail:
> > > > httpclient-users-help@hc.apache.org
> > > > 
> > > > 
> > > 
> > 
> > 
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail:
> > httpclient-users-help@hc.apache.org
> > 
> > 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by sreenivas somavarapu <so...@gmail.com>.
Hi Oleg,

My sample program (it's a kind of unit test for SharedOutputBuffer to
demonstrate that it is not able to write single byte if input byte array
size is greater than 1023) doesn't have flush as my intention is to show
the issue with SharedOutputBuffer alone and it is not a complete program
which sends the data to the actual server and get response back. Main issue
lies while writing data greater than 1023 bytes at first go itself
to SharedOutputBuffer from the byte array.

I think the below conditions in SharedOutputBuffer is causing this behavior
            while (src.hasRemaining()) {
                // always buffer small chunks
                *if (src.remaining() < 1024 && buffer().remaining() >
src.remaining()) {*
                    buffer().put(src);
                } else {
                   * if (buffer().position() > 0 || dataStreamChannel ==
null) {*
                        waitFlush();
                    }

Regards,
Sreenivas

On Mon, Mar 6, 2023 at 8:52 PM Oleg Kalnichevski <ol...@apache.org> wrote:

> On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
> > Hi Oleg,
> >
> > It looks like some issue with SharedOutputBuffer implementation and
> > it is
> > also not expandable as we have to allocate complete size during
> > initialization itself. If I change below two lines from my program it
> > proceeds further and completes the request (Basically
> > SharedOutputBuffer is
> > not able to write 1024 or above bytes of data at single go (If we
> > declare
> > temp block size as 1023 also it works)). It looks like there is an
> > explicit 1024 length check in SharedOutputBuffer which is causing
> > this
> > behavior.
> >
> > SharedOutputBuffer buffer = new
> > SharedOutputBuffer(req.contentLength() +
> > 1024);
>
> What you are doing is _completely_ wrong. SharedOutputBuffer is not
> meant to be expandable.
>
> You cannot just write into the buffer and expect it to work by magic.
> You also must flush the buffer.
>
> Oleg
>
> >
> > byte[] tmp = new byte[512];
> >
> > Earlier those 2 lines were as below
> >
> >  SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> >
> > byte[] tmp = new byte[4 * 1024];
> >
> > Below is my sample program on SharedOutputBuffer  which demonstrated
> > this
> > behavior
> >
> >         final StringBuilder sbuffer = new StringBuilder("123");
> >         System.out.println("O/P: " + sbuffer.toString());
> >         for (long i = 0; i < (64 * 1000); ++i) {
> >             sbuffer.append('A');
> >         }
> >
> >         String postData = sbuffer.toString();
> >         System.out.println("O/P1: " + postData);
> >
> >         byte[] inputData = postData.getBytes(StandardCharsets.UTF_8);
> >         ByteArrayInputStream inputStream = new
> > ByteArrayInputStream(inputData);
> >         SharedOutputBuffer outputStream = new SharedOutputBuffer(100
> > *
> > 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
> >         *byte[] tmp = new byte[1024];  <-- If we change this to 1023
> > then
> > it works*
> >         int l = -1;
> >         while ((l = inputStream.read(tmp)) != -1) {
> >             System.out.println("Writing Data");
> >             System.out.println(l);
> >             outputStream.write(tmp, 0, l);
> >             System.out.println("Wrote Data");
> >         }
> >
> >         System.out.println("Done Writing Data");
> >         outputStream.writeCompleted();
> >         System.out.println(outputStream.toString());
> >
> > Regards,
> > Sreenivas
> >
> > On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski <ol...@apache.org>
> > wrote:
> >
> > > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > > > Hi Team,
> > > >
> > > > We are using AbstractClassicEntityProducer with a customization
> > > > of
> > > > storing
> > > > / computing contentLength as well which works fine if we have
> > > > small
> > > > POST
> > > > data, but when POST data is big (for example 64 KB), it seems
> > > > OutPutstream
> > > > (internally using ContentOutputStream a wrapper over
> > > > SharedOutputBuffer of
> > > > initial size 1024) is not able to write data (We are using
> > > > temporary
> > > > buffer
> > > > block of 4kb but still it is not able to write 4KB of data). Is
> > > > there
> > > > any
> > > > limitation or some configuration problem?
> > > >
> > > > *Code being used*
> > > >         SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> > > >
> > > >         produceData(contentType, new
> > > > ContentOutputStream(buffer));
> > > >
> > > > protected void produceData(ContentType contentType, OutputStream
> > > > outputStream) throws IOException {
> > > >         byte[] tmp = new byte[4 * 1024];
> > > >         int l = -1;
> > > >         while ((l = inputStream.read(tmp)) != -1) {
> > > > *            outputStream.write(tmp, 0, l);  <---- This is where
> > > > request
> > > > hangs / waits*
> > >
> > > This is likely due to a bug in your code that prevents the shared
> > > buffer from being flushed.
> > >
> > > Oleg
> > >
> > >
> > > -------------------------------------------------------------------
> > > --
> > > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > > For additional commands, e-mail:
> > > httpclient-users-help@hc.apache.org
> > >
> > >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

-- 
Cheers,
S. Sreenivas

Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2023-03-06 at 20:20 +0530, sreenivas somavarapu wrote:
> Hi Oleg,
> 
> It looks like some issue with SharedOutputBuffer implementation and
> it is
> also not expandable as we have to allocate complete size during
> initialization itself. If I change below two lines from my program it
> proceeds further and completes the request (Basically
> SharedOutputBuffer is
> not able to write 1024 or above bytes of data at single go (If we
> declare
> temp block size as 1023 also it works)). It looks like there is an
> explicit 1024 length check in SharedOutputBuffer which is causing
> this
> behavior.
> 
> SharedOutputBuffer buffer = new
> SharedOutputBuffer(req.contentLength() +
> 1024);

What you are doing is _completely_ wrong. SharedOutputBuffer is not
meant to be expandable.

You cannot just write into the buffer and expect it to work by magic.
You also must flush the buffer.

Oleg

> 
> byte[] tmp = new byte[512];
> 
> Earlier those 2 lines were as below
> 
>  SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> 
> byte[] tmp = new byte[4 * 1024];
> 
> Below is my sample program on SharedOutputBuffer  which demonstrated
> this
> behavior
> 
>         final StringBuilder sbuffer = new StringBuilder("123");
>         System.out.println("O/P: " + sbuffer.toString());
>         for (long i = 0; i < (64 * 1000); ++i) {
>             sbuffer.append('A');
>         }
> 
>         String postData = sbuffer.toString();
>         System.out.println("O/P1: " + postData);
> 
>         byte[] inputData = postData.getBytes(StandardCharsets.UTF_8);
>         ByteArrayInputStream inputStream = new
> ByteArrayInputStream(inputData);
>         SharedOutputBuffer outputStream = new SharedOutputBuffer(100
> *
> 1024); *<-- If we keep this as 4 * 1024 also it hangs.*
>         *byte[] tmp = new byte[1024];  <-- If we change this to 1023
> then
> it works*
>         int l = -1;
>         while ((l = inputStream.read(tmp)) != -1) {
>             System.out.println("Writing Data");
>             System.out.println(l);
>             outputStream.write(tmp, 0, l);
>             System.out.println("Wrote Data");
>         }
> 
>         System.out.println("Done Writing Data");
>         outputStream.writeCompleted();
>         System.out.println(outputStream.toString());
> 
> Regards,
> Sreenivas
> 
> On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski <ol...@apache.org>
> wrote:
> 
> > On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > > Hi Team,
> > > 
> > > We are using AbstractClassicEntityProducer with a customization
> > > of
> > > storing
> > > / computing contentLength as well which works fine if we have
> > > small
> > > POST
> > > data, but when POST data is big (for example 64 KB), it seems
> > > OutPutstream
> > > (internally using ContentOutputStream a wrapper over
> > > SharedOutputBuffer of
> > > initial size 1024) is not able to write data (We are using
> > > temporary
> > > buffer
> > > block of 4kb but still it is not able to write 4KB of data). Is
> > > there
> > > any
> > > limitation or some configuration problem?
> > > 
> > > *Code being used*
> > >         SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> > > 
> > >         produceData(contentType, new
> > > ContentOutputStream(buffer));
> > > 
> > > protected void produceData(ContentType contentType, OutputStream
> > > outputStream) throws IOException {
> > >         byte[] tmp = new byte[4 * 1024];
> > >         int l = -1;
> > >         while ((l = inputStream.read(tmp)) != -1) {
> > > *            outputStream.write(tmp, 0, l);  <---- This is where
> > > request
> > > hangs / waits*
> > 
> > This is likely due to a bug in your code that prevents the shared
> > buffer from being flushed.
> > 
> > Oleg
> > 
> > 
> > -------------------------------------------------------------------
> > --
> > To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> > For additional commands, e-mail:
> > httpclient-users-help@hc.apache.org
> > 
> > 
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org


Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by sreenivas somavarapu <so...@gmail.com>.
Hi Oleg,

It looks like some issue with SharedOutputBuffer implementation and it is
also not expandable as we have to allocate complete size during
initialization itself. If I change below two lines from my program it
proceeds further and completes the request (Basically SharedOutputBuffer is
not able to write 1024 or above bytes of data at single go (If we declare
temp block size as 1023 also it works)). It looks like there is an
explicit 1024 length check in SharedOutputBuffer which is causing this
behavior.

SharedOutputBuffer buffer = new SharedOutputBuffer(req.contentLength() +
1024);

byte[] tmp = new byte[512];

Earlier those 2 lines were as below

 SharedOutputBuffer buffer = new SharedOutputBuffer(1024);

byte[] tmp = new byte[4 * 1024];

Below is my sample program on SharedOutputBuffer  which demonstrated this
behavior

        final StringBuilder sbuffer = new StringBuilder("123");
        System.out.println("O/P: " + sbuffer.toString());
        for (long i = 0; i < (64 * 1000); ++i) {
            sbuffer.append('A');
        }

        String postData = sbuffer.toString();
        System.out.println("O/P1: " + postData);

        byte[] inputData = postData.getBytes(StandardCharsets.UTF_8);
        ByteArrayInputStream inputStream = new
ByteArrayInputStream(inputData);
        SharedOutputBuffer outputStream = new SharedOutputBuffer(100 *
1024); *<-- If we keep this as 4 * 1024 also it hangs.*
        *byte[] tmp = new byte[1024];  <-- If we change this to 1023 then
it works*
        int l = -1;
        while ((l = inputStream.read(tmp)) != -1) {
            System.out.println("Writing Data");
            System.out.println(l);
            outputStream.write(tmp, 0, l);
            System.out.println("Wrote Data");
        }

        System.out.println("Done Writing Data");
        outputStream.writeCompleted();
        System.out.println(outputStream.toString());

Regards,
Sreenivas

On Mon, Mar 6, 2023 at 7:17 PM Oleg Kalnichevski <ol...@apache.org> wrote:

> On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> > Hi Team,
> >
> > We are using AbstractClassicEntityProducer with a customization of
> > storing
> > / computing contentLength as well which works fine if we have small
> > POST
> > data, but when POST data is big (for example 64 KB), it seems
> > OutPutstream
> > (internally using ContentOutputStream a wrapper over
> > SharedOutputBuffer of
> > initial size 1024) is not able to write data (We are using temporary
> > buffer
> > block of 4kb but still it is not able to write 4KB of data). Is there
> > any
> > limitation or some configuration problem?
> >
> > *Code being used*
> >         SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> >
> >         produceData(contentType, new ContentOutputStream(buffer));
> >
> > protected void produceData(ContentType contentType, OutputStream
> > outputStream) throws IOException {
> >         byte[] tmp = new byte[4 * 1024];
> >         int l = -1;
> >         while ((l = inputStream.read(tmp)) != -1) {
> > *            outputStream.write(tmp, 0, l);  <---- This is where
> > request
> > hangs / waits*
>
> This is likely due to a bug in your code that prevents the shared
> buffer from being flushed.
>
> Oleg
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
> For additional commands, e-mail: httpclient-users-help@hc.apache.org
>
>

-- 
Cheers,
S. Sreenivas

Re: AbstractClassicEntityProducer usage and big POST data around 64kb seems hanging

Posted by Oleg Kalnichevski <ol...@apache.org>.
On Mon, 2023-03-06 at 16:26 +0530, sreenivas somavarapu wrote:
> Hi Team,
> 
> We are using AbstractClassicEntityProducer with a customization of
> storing
> / computing contentLength as well which works fine if we have small
> POST
> data, but when POST data is big (for example 64 KB), it seems
> OutPutstream
> (internally using ContentOutputStream a wrapper over
> SharedOutputBuffer of
> initial size 1024) is not able to write data (We are using temporary
> buffer
> block of 4kb but still it is not able to write 4KB of data). Is there
> any
> limitation or some configuration problem?
> 
> *Code being used*
>         SharedOutputBuffer buffer = new SharedOutputBuffer(1024);
> 
>         produceData(contentType, new ContentOutputStream(buffer));
> 
> protected void produceData(ContentType contentType, OutputStream
> outputStream) throws IOException {
>         byte[] tmp = new byte[4 * 1024];
>         int l = -1;
>         while ((l = inputStream.read(tmp)) != -1) {
> *            outputStream.write(tmp, 0, l);  <---- This is where
> request
> hangs / waits*

This is likely due to a bug in your code that prevents the shared
buffer from being flushed.

Oleg


---------------------------------------------------------------------
To unsubscribe, e-mail: httpclient-users-unsubscribe@hc.apache.org
For additional commands, e-mail: httpclient-users-help@hc.apache.org