You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Bernard Ligny <be...@gmail.com> on 2013/07/02 11:10:11 UTC

Chunking issue with http producer

I've got a back-end web service returning large **chunked** data.
I have defined a Camel route with a proxy to this web service.
I do not want Camel to buffer the entire back-end response, and then return
it as a whole to the client.
Instead, i would like to preserve the original streaming (mainly to avoid
read timeouts on the client side).

How can i achieve this (a "pipe" actually) in Camel ?
Even when I disable "streamCache" on CamelContext, it seems Camel is yet
buffering internally the response.

Hereunder the difference between a direct call to the back-end, and the same
call via Camel:

<http://camel.465427.n5.nabble.com/file/n5735075/difference.png> 

=> see Latency !
In the first case, client gets a beginning of response after 2 sec,
while in the second case, he has to wait for 12 sec before getting any
data...



I looked into the code of "HttpProducer.java", and I'm a bit puzzled about
what i can see in the "doExtractResponseBodyAsStream" method:


private static InputStream doExtractResponseBodyAsStream(InputStream is,
Exchange exchange) throws IOException {
        // As httpclient is using a AutoCloseInputStream, it will be closed
when the connection is closed
        // we need to cache the stream for it.
        try {
            // This CachedOutputStream will not be closed when the exchange
is onCompletion
            CachedOutputStream cos = new CachedOutputStream(exchange,
false);
            IOHelper.copy(is, cos);
            // When the InputStream is closed, the CachedOutputStream will
be closed
            return cos.getWrappedInputStream();
        } finally {
            IOHelper.close(is, "Extracting response body", LOG);
        }
    }


If you always *copy* the stream, you need to read the whole stream :-(
Where do you take account of the "streamCache" flag when deciding to cache
or not cache the response input stream ?

Thanks for your help.

Bernard.





--
View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Chunking issue with http producer

Posted by "Willem.Jiang" <wi...@gmail.com>.
Hi Bernard,

You are right, the patch has nothing to do with if the http endpoint is
bridge endpoint or not. 
I created a JIRA[1] to keep track of this useful feature.

[1]https://issues.apache.org/jira/browse/CAMEL-7638

Regards,

Willem



--
View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075p5754439.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Chunking issue with http producer

Posted by Bernard Ligny <be...@gmail.com>.
Can you please explain why in "Bridge mode *only*" ?

>From the doc:
"bridgeEndpoint : HttpProducer will ignore the Exchange.HTTP_URI header, and
use the endpoint's URI for request"

Sorry, but I do the not understand the concrete consequences of using URI x
instead of URI y on the opportunity to stream or cache http response...

I have tested it on a non-bridged endpoint, and it is working very well.


Willem.Jiang wrote
> I think we can enable this kind of feature when the http producer is
> working in the Bridge mode.





--
View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075p5735142.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Chunking issue with http producer

Posted by Willem jiang <wi...@gmail.com>.
Thanks for sharing the solution with us.
I think we can enable this kind of feature when the http producer is working in the Bridge mode.


--  
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem





On Wednesday, July 3, 2013 at 5:47 PM, Bernard Ligny wrote:

> I finally found a solution that i am sharing here for the community.
> With that proposed patch, the streaming capability is preserved, and the
> latency is now much better:
>  
> <http://camel.465427.n5.nabble.com/file/n5735129/difference2.png>  
>  
>  
> Here are the details of my patching of "*HttpProducer.java*"
>  
>  
>  
> There is one small drawback: since the http connection is released only when
> the stream is closed, it is very important to...close the stream ;-)
> (NB: compared to AutoCloseInputStream, which automatically "/closes and
> discards the underlying stream as soon as the end of input has been
> reached/"...)
>  
> Yet interested by your opinion about other potential drawbacks...
>  
> Bernard.
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075p5735129.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).




Re: Chunking issue with http producer

Posted by Bernard Ligny <be...@gmail.com>.
I finally found a solution that i am sharing here for the community.
With that proposed patch, the streaming capability is preserved, and the
latency is now much better:

<http://camel.465427.n5.nabble.com/file/n5735129/difference2.png> 


Here are the details of my patching of "*HttpProducer.java*"



There is one small drawback: since the http connection is released only when
the stream is closed, it is very important to...close the stream ;-)
(NB: compared to AutoCloseInputStream, which automatically "/closes and
discards the underlying stream as soon as the end of input has been
reached/"...)

Yet interested by your opinion about other potential drawbacks...

Bernard.



--
View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075p5735129.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Chunking issue with http producer

Posted by Bernard Ligny <be...@gmail.com>.
Thanks for your response.

Wouldn't it be interesting for next releases to have an option to (not cache
and) *defer* the connection closing till the exchange (that is, behind the
scene, the input stream of http response) is *effectively* (fully) read ?
I am thinking for instance about using a completion callback (via
Exchange.addOnCompletion() method) doing the close on that moment...

Bernard.




--
View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075p5735127.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Re: Chunking issue with http producer

Posted by Willem jiang <wi...@gmail.com>.
If you are using http client 3, it will clean up the input stream when you close the connection.  
You can find more information by checking the the comments in the doExtractResponseBodyAsStream.

That is why we need to cache the input stream as it will be used in the other part of camel route.

If you disable the streamCache it doesn't affect it, but you may consider to disable cache the stream into file as the default size is 64K.  

--  
Willem Jiang

Red Hat, Inc.
FuseSource is now part of Red Hat
Web: http://www.fusesource.com | http://www.redhat.com
Blog: http://willemjiang.blogspot.com (http://willemjiang.blogspot.com/) (English)
          http://jnn.iteye.com (http://jnn.javaeye.com/) (Chinese)
Twitter: willemjiang  
Weibo: 姜宁willem





On Tuesday, July 2, 2013 at 5:10 PM, Bernard Ligny wrote:

> I've got a back-end web service returning large **chunked** data.
> I have defined a Camel route with a proxy to this web service.
> I do not want Camel to buffer the entire back-end response, and then return
> it as a whole to the client.
> Instead, i would like to preserve the original streaming (mainly to avoid
> read timeouts on the client side).
>  
> How can i achieve this (a "pipe" actually) in Camel ?
> Even when I disable "streamCache" on CamelContext, it seems Camel is yet
> buffering internally the response.
>  
> Hereunder the difference between a direct call to the back-end, and the same
> call via Camel:
>  
> <http://camel.465427.n5.nabble.com/file/n5735075/difference.png>  
>  
> => see Latency !
> In the first case, client gets a beginning of response after 2 sec,
> while in the second case, he has to wait for 12 sec before getting any
> data...
>  
>  
>  
> I looked into the code of "HttpProducer.java", and I'm a bit puzzled about
> what i can see in the "doExtractResponseBodyAsStream" method:
>  
>  
> private static InputStream doExtractResponseBodyAsStream(InputStream is,
> Exchange exchange) throws IOException {
> // As httpclient is using a AutoCloseInputStream, it will be closed
> when the connection is closed
> // we need to cache the stream for it.
> try {
> // This CachedOutputStream will not be closed when the exchange
> is onCompletion
> CachedOutputStream cos = new CachedOutputStream(exchange,
> false);
> IOHelper.copy(is, cos);
> // When the InputStream is closed, the CachedOutputStream will
> be closed
> return cos.getWrappedInputStream();
> } finally {
> IOHelper.close(is, "Extracting response body", LOG);
> }
> }
>  
>  
> If you always *copy* the stream, you need to read the whole stream :-(
> Where do you take account of the "streamCache" flag when deciding to cache
> or not cache the response input stream ?
>  
> Thanks for your help.
>  
> Bernard.
>  
>  
>  
>  
>  
> --
> View this message in context: http://camel.465427.n5.nabble.com/Chunking-issue-with-http-producer-tp5735075.html
> Sent from the Camel - Users mailing list archive at Nabble.com (http://Nabble.com).