You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by J- <jm...@hotmail.com> on 2016/05/13 10:03:52 UTC
Restlet GET request can cause Memeory leak/Stream not closed due to
CAMEL-9611
My apologies, this bug is 100% my fault because of CAMEL-9611.
Jira is in some weird locked down state so i can't post the issue on there, but this is pretty major.
The fact that we no longer read the message body for Restlet GET request has created a very painful bug (that took me 3 days to figure out).
The problem is that if body is never read and the body is a connection/stream, it is never closed.
To reproduce just call restlet GET 20 times in a row.
from("direct:start").loop(20).to("restlet:http://<anyURL>?restletMethod=get");
Simpiliest fix for me was to read the body and dump it for the GET path in DefaultRestletBinding.java
@line 220:
else {
// no body
// dump body to close streams
try{
String garbage = exchange.getIn().getBody(String.class);
}catch(Exception ex){
//we don't care if this fails
}
LOG.debug("Populate Restlet {} request from exchange using media type {}", method, mediaType);
request.setEntity(new EmptyRepresentation());
}
Re: Restlet GET request can cause Memeory leak/Stream not closed due
to CAMEL-9611
Posted by J- <jm...@hotmail.com>.
The problem with our flow was
restlet:<someURL> -> if response header = '200' (body never read) call -> restlet:<anotherURL>
After around 20 successful calls it broke.
If you open a Jira can you send the Jira# so we can track it.
Thanks.
________________________________________
From: Claus Ibsen <cl...@gmail.com>
Sent: Friday, May 13, 2016 10:04 AM
To: users@camel.apache.org
Subject: Re: Restlet GET request can cause Memeory leak/Stream not closed due to CAMEL-9611
If you do a loop like that then convert the body to a string first etc
loop(20).convertBodyTo(String.class)
the problem is that in the loop you use the response from previous
restlet call as input for the next, and so on.
And as you say since the input is not read, then you need to convert
to string, to cause it to read the stream so its closed by restlet
itself.
However we should also tell Camel to auto close the stream when the
exchange is done, so I am adding that.
On Fri, May 13, 2016 at 3:40 PM, J- <jm...@hotmail.com> wrote:
> Just do this quick test:
>
> from("direct:start").loop(20).to("restlet:http://<anyURL>");
>
> you will get this exception because connections aren't closed until they are read from:
>
> org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:412)
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298)
> at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at org.restlet.ext.httpclient.internal.HttpMethodCall.sendRequest(HttpMethodCall.java:339)
> at org.restlet.engine.adapter.ClientAdapter.commit(ClientAdapter.java:105)
> at org.restlet.engine.adapter.HttpClientHelper.handle(HttpClientHelper.java:119)
> at org.restlet.Client.handle(Client.java:153)
> at org.restlet.Restlet.handle(Restlet.java:275)
> at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:79)
> at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:98)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:128)
> at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:96)
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
>
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
>
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
>
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
>
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
>
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
>
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:138)
>
> at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:109)
>
> at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
> at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)
>
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
>
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:298)
>
> at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)
>
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
>
> at java.lang.Thread.run(Thread.java:745)
>
> ________________________________________
> From: Claus Ibsen <cl...@gmail.com>
> Sent: Friday, May 13, 2016 7:55 AM
> To: users@camel.apache.org
> Subject: Re: Restlet GET request can cause Memeory leak/Stream not closed due to CAMEL-9611
>
> Hi
>
> It seems wrong what you are doing as a workaround.
>
> You must have something that creates the message body as a
> connection/stream that should be responsible for closing it when its
> done being routed in Camel. So it smells like you should look at the
> route consumer.
>
> What are you using?
>
> On Fri, May 13, 2016 at 12:03 PM, J- <jm...@hotmail.com> wrote:
>> My apologies, this bug is 100% my fault because of CAMEL-9611.
>> Jira is in some weird locked down state so i can't post the issue on there, but this is pretty major.
>>
>> The fact that we no longer read the message body for Restlet GET request has created a very painful bug (that took me 3 days to figure out).
>> The problem is that if body is never read and the body is a connection/stream, it is never closed.
>>
>> To reproduce just call restlet GET 20 times in a row.
>>
>> from("direct:start").loop(20).to("restlet:http://<anyURL>?restletMethod=get");
>>
>> Simpiliest fix for me was to read the body and dump it for the GET path in DefaultRestletBinding.java
>>
>> @line 220:
>> else {
>> // no body
>> // dump body to close streams
>> try{
>> String garbage = exchange.getIn().getBody(String.class);
>> }catch(Exception ex){
>> //we don't care if this fails
>> }
>> LOG.debug("Populate Restlet {} request from exchange using media type {}", method, mediaType);
>> request.setEntity(new EmptyRepresentation());
>> }
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Restlet GET request can cause Memeory leak/Stream not closed due
to CAMEL-9611
Posted by Claus Ibsen <cl...@gmail.com>.
If you do a loop like that then convert the body to a string first etc
loop(20).convertBodyTo(String.class)
the problem is that in the loop you use the response from previous
restlet call as input for the next, and so on.
And as you say since the input is not read, then you need to convert
to string, to cause it to read the stream so its closed by restlet
itself.
However we should also tell Camel to auto close the stream when the
exchange is done, so I am adding that.
On Fri, May 13, 2016 at 3:40 PM, J- <jm...@hotmail.com> wrote:
> Just do this quick test:
>
> from("direct:start").loop(20).to("restlet:http://<anyURL>");
>
> you will get this exception because connections aren't closed until they are read from:
>
> org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:412)
> at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298)
> at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238)
> at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
> at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
> at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
> at org.restlet.ext.httpclient.internal.HttpMethodCall.sendRequest(HttpMethodCall.java:339)
> at org.restlet.engine.adapter.ClientAdapter.commit(ClientAdapter.java:105)
> at org.restlet.engine.adapter.HttpClientHelper.handle(HttpClientHelper.java:119)
> at org.restlet.Client.handle(Client.java:153)
> at org.restlet.Restlet.handle(Restlet.java:275)
> at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:79)
> at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:98)
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:128)
> at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:96)
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
>
> at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
>
> at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
>
> at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
>
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
>
> at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
>
> at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
>
> at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:138)
>
> at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:109)
>
> at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
> at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)
>
> at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
>
> at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)
>
> at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
>
> at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:298)
>
> at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)
>
> at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
>
> at java.lang.Thread.run(Thread.java:745)
>
> ________________________________________
> From: Claus Ibsen <cl...@gmail.com>
> Sent: Friday, May 13, 2016 7:55 AM
> To: users@camel.apache.org
> Subject: Re: Restlet GET request can cause Memeory leak/Stream not closed due to CAMEL-9611
>
> Hi
>
> It seems wrong what you are doing as a workaround.
>
> You must have something that creates the message body as a
> connection/stream that should be responsible for closing it when its
> done being routed in Camel. So it smells like you should look at the
> route consumer.
>
> What are you using?
>
> On Fri, May 13, 2016 at 12:03 PM, J- <jm...@hotmail.com> wrote:
>> My apologies, this bug is 100% my fault because of CAMEL-9611.
>> Jira is in some weird locked down state so i can't post the issue on there, but this is pretty major.
>>
>> The fact that we no longer read the message body for Restlet GET request has created a very painful bug (that took me 3 days to figure out).
>> The problem is that if body is never read and the body is a connection/stream, it is never closed.
>>
>> To reproduce just call restlet GET 20 times in a row.
>>
>> from("direct:start").loop(20).to("restlet:http://<anyURL>?restletMethod=get");
>>
>> Simpiliest fix for me was to read the body and dump it for the GET path in DefaultRestletBinding.java
>>
>> @line 220:
>> else {
>> // no body
>> // dump body to close streams
>> try{
>> String garbage = exchange.getIn().getBody(String.class);
>> }catch(Exception ex){
>> //we don't care if this fails
>> }
>> LOG.debug("Populate Restlet {} request from exchange using media type {}", method, mediaType);
>> request.setEntity(new EmptyRepresentation());
>> }
>
>
>
> --
> Claus Ibsen
> -----------------
> http://davsclaus.com @davsclaus
> Camel in Action 2: https://www.manning.com/ibsen2
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Restlet GET request can cause Memeory leak/Stream not closed due
to CAMEL-9611
Posted by J- <jm...@hotmail.com>.
Just do this quick test:
from("direct:start").loop(20).to("restlet:http://<anyURL>");
you will get this exception because connections aren't closed until they are read from:
org.apache.http.conn.ConnectionPoolTimeoutException: Timeout waiting for connection from pool
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute.getEntryBlocking(ConnPoolByRoute.java:412)
at org.apache.http.impl.conn.tsccm.ConnPoolByRoute$1.getPoolEntry(ConnPoolByRoute.java:298)
at org.apache.http.impl.conn.tsccm.ThreadSafeClientConnManager$1.getConnection(ThreadSafeClientConnManager.java:238)
at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:423)
at org.apache.http.impl.client.AbstractHttpClient.doExecute(AbstractHttpClient.java:882)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:82)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:107)
at org.apache.http.impl.client.CloseableHttpClient.execute(CloseableHttpClient.java:55)
at org.restlet.ext.httpclient.internal.HttpMethodCall.sendRequest(HttpMethodCall.java:339)
at org.restlet.engine.adapter.ClientAdapter.commit(ClientAdapter.java:105)
at org.restlet.engine.adapter.HttpClientHelper.handle(HttpClientHelper.java:119)
at org.restlet.Client.handle(Client.java:153)
at org.restlet.Restlet.handle(Restlet.java:275)
at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:79)
at org.apache.camel.component.restlet.RestletProducer.process(RestletProducer.java:98)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:128)
at org.apache.camel.processor.LoopProcessor.process(LoopProcessor.java:96)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.direct.DirectProducer.process(DirectProducer.java:62)
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:145)
at org.apache.camel.management.InstrumentationProcessor.process(InstrumentationProcessor.java:77)
at org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:468)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:121)
at org.apache.camel.processor.Pipeline.process(Pipeline.java:83)
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:190)
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.processAsynchronously(ServerChannelHandler.java:138)
at org.apache.camel.component.netty4.handlers.ServerChannelHandler.channelRead0(ServerChannelHandler.java:109)
at org.apache.camel.component.netty4.http.handlers.HttpServerChannelHandler.channelRead0(HttpServerChannelHandler.java:211)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at org.apache.camel.component.netty4.http.handlers.HttpServerMultiplexChannelHandler.channelRead0(HttpServerMultiplexChannelHandler.java:113)
at io.netty.channel.SimpleChannelInboundHandler.channelRead(SimpleChannelInboundHandler.java:105)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:307)
at io.netty.channel.AbstractChannelHandlerContext.access$600(AbstractChannelHandlerContext.java:32)
at io.netty.channel.AbstractChannelHandlerContext$7.run(AbstractChannelHandlerContext.java:298)
at io.netty.util.concurrent.DefaultEventExecutor.run(DefaultEventExecutor.java:36)
at io.netty.util.concurrent.SingleThreadEventExecutor$2.run(SingleThreadEventExecutor.java:112)
at java.lang.Thread.run(Thread.java:745)
________________________________________
From: Claus Ibsen <cl...@gmail.com>
Sent: Friday, May 13, 2016 7:55 AM
To: users@camel.apache.org
Subject: Re: Restlet GET request can cause Memeory leak/Stream not closed due to CAMEL-9611
Hi
It seems wrong what you are doing as a workaround.
You must have something that creates the message body as a
connection/stream that should be responsible for closing it when its
done being routed in Camel. So it smells like you should look at the
route consumer.
What are you using?
On Fri, May 13, 2016 at 12:03 PM, J- <jm...@hotmail.com> wrote:
> My apologies, this bug is 100% my fault because of CAMEL-9611.
> Jira is in some weird locked down state so i can't post the issue on there, but this is pretty major.
>
> The fact that we no longer read the message body for Restlet GET request has created a very painful bug (that took me 3 days to figure out).
> The problem is that if body is never read and the body is a connection/stream, it is never closed.
>
> To reproduce just call restlet GET 20 times in a row.
>
> from("direct:start").loop(20).to("restlet:http://<anyURL>?restletMethod=get");
>
> Simpiliest fix for me was to read the body and dump it for the GET path in DefaultRestletBinding.java
>
> @line 220:
> else {
> // no body
> // dump body to close streams
> try{
> String garbage = exchange.getIn().getBody(String.class);
> }catch(Exception ex){
> //we don't care if this fails
> }
> LOG.debug("Populate Restlet {} request from exchange using media type {}", method, mediaType);
> request.setEntity(new EmptyRepresentation());
> }
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2
Re: Restlet GET request can cause Memeory leak/Stream not closed due
to CAMEL-9611
Posted by Claus Ibsen <cl...@gmail.com>.
Hi
It seems wrong what you are doing as a workaround.
You must have something that creates the message body as a
connection/stream that should be responsible for closing it when its
done being routed in Camel. So it smells like you should look at the
route consumer.
What are you using?
On Fri, May 13, 2016 at 12:03 PM, J- <jm...@hotmail.com> wrote:
> My apologies, this bug is 100% my fault because of CAMEL-9611.
> Jira is in some weird locked down state so i can't post the issue on there, but this is pretty major.
>
> The fact that we no longer read the message body for Restlet GET request has created a very painful bug (that took me 3 days to figure out).
> The problem is that if body is never read and the body is a connection/stream, it is never closed.
>
> To reproduce just call restlet GET 20 times in a row.
>
> from("direct:start").loop(20).to("restlet:http://<anyURL>?restletMethod=get");
>
> Simpiliest fix for me was to read the body and dump it for the GET path in DefaultRestletBinding.java
>
> @line 220:
> else {
> // no body
> // dump body to close streams
> try{
> String garbage = exchange.getIn().getBody(String.class);
> }catch(Exception ex){
> //we don't care if this fails
> }
> LOG.debug("Populate Restlet {} request from exchange using media type {}", method, mediaType);
> request.setEntity(new EmptyRepresentation());
> }
--
Claus Ibsen
-----------------
http://davsclaus.com @davsclaus
Camel in Action 2: https://www.manning.com/ibsen2