I have a stand-alone Java 1.7 application using *Camel 2.15.2* and the *camel-netty* component which is running on Linux RHEL6. In production, with a camel route that consumes via *netty:udp*, we appear to be facing an issue similar to this old one: ChannelPool closing channels prematurely <https://linkedin.jira.com/plugins/servlet/mobile#issue/NOR-19> *QUESTION:* When using *camel:netty*, is there a way to set netty's *closeChannelTimeMillis* option?, I don't see it listed on the camel-netty <http://camel.apache.org/netty.html> page. I'm hoping that there might be an undocumented or back-door way to set it. Thanks, SteveR -- View this message in context: http://camel.465427.n5.nabble.com/camel-netty-How-to-set-the-netty-closeChannelTimeMillis-option-tp5772342.html Sent from the Camel - Users mailing list archive at Nabble.com.
It looks like you need to setup the sync option to be true when wiretapping the message.
--
Willem Jiang
Blog: http://willemjiang.blogspot.com (English)
http://jnn.iteye.com (Chinese)
Twitter: willemjiang
Weibo: 姜宁willem
On October 8, 2015 at 12:52:20 PM, SteveR (srichardson@vonage.com) wrote:
> Hi Willem:
>
> Thanks for taking the time to reply. Actually, we have not been specifying
> the camel-netty *udpConnectionlessSending=true *option, and I see that the
> default is *false* if not explicitly specified.
>
> We were actually contemplating to set *udpConnectionlessSending=true*,
> thinking that it might help.
>
> The Camel route in question is shown below. The issue we are having in
> production, is that we see a flooding of the below ERROR messages in our
> Camel *rootLog.txt* which indicates a problem with *ClosedChannelException
> *when trying to mirror the received UDP packets to a remote host via the
> Camel wireTap() shown in the route below. Note also, that the remote host
> attempts to ACKknowledge each received UDP packet that this Camel route
> mirrors to it. So I hope those ACKs do not cause any strange side effects
> to the Camel route. We're stuck on exactly how to go about debugging this
> issue. So the wireTap portion of the route seems to mostly be failing.
> However, some UDP packets do get through to the remote host.
>
> Any thoughts are greatly appreciated.
>
> Thanks, SteveR
>
>
> > xmlns="http://camel.apache.org/schema/spring">
> > uri="netty:udp://talend-es-edge-01.kewr0.s.missionnetworks.net:11111?serverPipelineFactory=#CQMS_SERVER_PIPELINE_FACTORY_ROUTE_ID_RAW_CQMS_EVENTS&keepAlive=true&sync=true&orderedThreadPoolExecutor=false&receiveBufferSize=8388608&sendBufferSize=8388608&allowDefaultCodec=false&disconnectOnNoReply=false&receiveBufferSizePredictor=8192"/>
>
> iso-8859-1
>
> > threadName="threads_ROUTE_ID_RAW_CQMS_EVENTS" callerRunsWhenRejected="true"
> id="threads1">
> > uri="netty:udp://abinitio-edge-01.kewr0.s.missionnetworks.net:11111?clientPipelineFactory=#CQMS_CLIENT_PIPELINE_FACTORY_ROUTE_ID_RAW_CQMS_EVENTS&sync=false&sendBufferSize=8388608&allowDefaultCodec=false&orderedThreadPoolExecutor=false"
> customId="true" id="ROUTE_ID_RAW_CQMS_EVENTS_MIRROR"/>
>
> > uri="seda:SEDA_TOP_102?size=100000&concurrentConsumers=10&waitForTaskToComplete=Never&failIfNoConsumers=true&timeout=10000"
> id="to1"/>
> > uri="seda:SEDA_ACK_103?size=100000&concurrentConsumers=10&waitForTaskToComplete=IfReplyExpected&failIfNoConsumers=true&timeout=10000"
> customId="true" id="ROUTE_ID_RAW_CQMS_EVENTS_TO_FIRST_URIS"/>
>
> iso-8859-1
>
>
>
>
>
> [2015-10-07 17:47:26,958] ERROR [Camel (MPLR_CQMS) thread #55 - WireTap]
> Failed delivery for (MessageId:
> ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342281
> on ExchangeId:
> ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342282).
> Exhausted after delivery attempt: 1 caught:
> java.nio.channels.ClosedChannelException
>
> Message History
>
> ---------------------------------------------------------------------------------------------------------------------------------------
> RouteId ProcessorId Processor
> Elapsed (ms)
> [ROUTE_ID_RAW_CQMS_] [ROUTE_ID_RAW_CQMS_]
> [udp://talend-es-edge-01.kewr0.s.missionnetworks.net:11111
> ] [ 1]
> [ROUTE_ID_RAW_CQMS_] [setProperty1 ] [setProperty[CamelCharsetName]
> ] [ 0]
> [ROUTE_ID_RAW_CQMS_] [threads1 ] [threads
> ] [ 1]
> [ROUTE_ID_RAW_CQMS_] [ROUTE_ID_RAW_CQMS_]
> [wireTap[netty:udp://abinitio-edge-01.kewr0.s.missionnetworks.net:11111?clientPi]
> [ 0]
>
> Exchange
>
> ---------------------------------------------------------------------------------------------------------------------------------------
> Exchange[
> Id
> ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342282
> ExchangePattern InOnly
> Headers
> {breadcrumbId=ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342277,
> CamelNettyChannelHandlerContext=org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5efa3b3c,
> CamelNettyLocalAddress=/69.59.252.44:11111, CamelNettyMessageEvent=[id:
> 0x5612eec3, /69.59.252.44:11111] RECEIVED: { "protocol":"MQDP-2",
> "CQM_STREAM_INFO": "P_RTCP_XRVM_COUNT":10} ] } } from
> /69.59.248.171:9999, CamelNettyRemoteAddress=/69.59.248.171:9999,
> CamelRedelivered=false, CamelRedeliveryCounter=0}
> BodyType String
> Body { "protocol":"MQDP-2", "CQM_STREAM_INFO":
> "P_RTCP_XRVM_COUNT":10} ] }}
> ]
>
> Stacktrace
>
> ---------------------------------------------------------------------------------------------------------------------------------------
>
> java.nio.channels.ClosedChannelException
> at
> org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.socket.nio.NioDatagramWorker.writeFromUserCode(NioDatagramWorker.java:212)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:97)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
> ~[netty-3.9.6.Final.jar:?]
> at org.jboss.netty.channel.Channels.write(Channels.java:725)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
> ~[netty-3.9.6.Final.jar:?]
> at org.jboss.netty.channel.Channels.write(Channels.java:704)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:252)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.jboss.netty.channel.socket.nio.NioDatagramChannel.write(NioDatagramChannel.java:299)
> ~[netty-3.9.6.Final.jar:?]
> at
> org.apache.camel.component.netty.NettyHelper.writeBodyAsync(NettyHelper.java:93)
> [camel-netty-2.15.2.jar:2.15.2]
> at
> org.apache.camel.component.netty.NettyProducer.process(NettyProducer.java:263)
> [camel-netty-2.15.2.jar:2.15.2]
> at
> org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
> [camel-core-2.15.2.jar:2.15.2]
> at
> org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
> [camel-core-2.15.2.jar:2.15.2]
> at
> org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
> [camel-core-2.15.2.jar:2.15.2]
> at
> org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
> [camel-core-2.15.2.jar:2.15.2]
> at
> org.apache.camel.processor.WireTapProcessor$1.call(WireTapProcessor.java:119)
> [camel-core-2.15.2.jar:2.15.2]
> at
> org.apache.camel.processor.WireTapProcessor$1.call(WireTapProcessor.java:113)
> [camel-core-2.15.2.jar:2.15.2]
> at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> [?:1.7.0_75]
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> [?:1.7.0_75]
> at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
>
>
>
>
>
>
> --
> View this message in context: http://camel.465427.n5.nabble.com/camel-netty-How-to-set-the-netty-closeChannelTimeMillis-option-tp5772342p5772394.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>
Hi Willem:
Thanks for taking the time to reply. Actually, we have not been specifying
the camel-netty *udpConnectionlessSending=true *option, and I see that the
default is *false* if not explicitly specified.
We were actually contemplating to set *udpConnectionlessSending=true*,
thinking that it might help.
The Camel route in question is shown below. The issue we are having in
production, is that we see a flooding of the below ERROR messages in our
Camel *rootLog.txt* which indicates a problem with *ClosedChannelException
*when trying to mirror the received UDP packets to a remote host via the
Camel wireTap() shown in the route below. Note also, that the remote host
attempts to ACKknowledge each received UDP packet that this Camel route
mirrors to it. So I hope those ACKs do not cause any strange side effects
to the Camel route. We're stuck on exactly how to go about debugging this
issue. So the wireTap portion of the route seems to mostly be failing.
However, some UDP packets do get through to the remote host.
Any thoughts are greatly appreciated.
Thanks, SteveR
<?xml version="1.0" encoding="UTF-8" standalone="yes"?>
<route startupOrder="104" customId="true" id="ROUTE_ID_RAW_CQMS_EVENTS"
xmlns="http://camel.apache.org/schema/spring">
<from
uri="netty:udp://talend-es-edge-01.kewr0.s.missionnetworks.net:11111?serverPipelineFactory=#CQMS_SERVER_PIPELINE_FACTORY_ROUTE_ID_RAW_CQMS_EVENTS&keepAlive=true&sync=true&orderedThreadPoolExecutor=false&receiveBufferSize=8388608&sendBufferSize=8388608&allowDefaultCodec=false&disconnectOnNoReply=false&receiveBufferSizePredictor=8192"/>
<setProperty propertyName="CamelCharsetName" id="setProperty1">
<expressionDefinition>iso-8859-1</expressionDefinition>
</setProperty>
<threads poolSize="7" maxPoolSize="14"
threadName="threads_ROUTE_ID_RAW_CQMS_EVENTS" callerRunsWhenRejected="true"
id="threads1">
<wireTap
uri="netty:udp://abinitio-edge-01.kewr0.s.missionnetworks.net:11111?clientPipelineFactory=#CQMS_CLIENT_PIPELINE_FACTORY_ROUTE_ID_RAW_CQMS_EVENTS&sync=false&sendBufferSize=8388608&allowDefaultCodec=false&orderedThreadPoolExecutor=false"
customId="true" id="ROUTE_ID_RAW_CQMS_EVENTS_MIRROR"/>
<multicast parallelProcessing="true" id="multicast1">
<to
uri="seda:SEDA_TOP_102?size=100000&concurrentConsumers=10&waitForTaskToComplete=Never&failIfNoConsumers=true&timeout=10000"
id="to1"/>
<to
uri="seda:SEDA_ACK_103?size=100000&concurrentConsumers=10&waitForTaskToComplete=IfReplyExpected&failIfNoConsumers=true&timeout=10000"
customId="true" id="ROUTE_ID_RAW_CQMS_EVENTS_TO_FIRST_URIS"/>
<setProperty propertyName="CamelCharsetName" id="setProperty2">
<expressionDefinition>iso-8859-1</expressionDefinition>
</setProperty>
</multicast>
</threads>
[2015-10-07 17:47:26,958] ERROR [Camel (MPLR_CQMS) thread #55 - WireTap]
Failed delivery for (MessageId:
ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342281
on ExchangeId:
ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342282).
Exhausted after delivery attempt: 1 caught:
java.nio.channels.ClosedChannelException
Message History
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor
Elapsed (ms)
[ROUTE_ID_RAW_CQMS_] [ROUTE_ID_RAW_CQMS_]
[udp://talend-es-edge-01.kewr0.s.missionnetworks.net:11111
] [ 1]
[ROUTE_ID_RAW_CQMS_] [setProperty1 ] [setProperty[CamelCharsetName]
] [ 0]
[ROUTE_ID_RAW_CQMS_] [threads1 ] [threads
] [ 1]
[ROUTE_ID_RAW_CQMS_] [ROUTE_ID_RAW_CQMS_]
[wireTap[netty:udp://abinitio-edge-01.kewr0.s.missionnetworks.net:11111?clientPi]
[ 0]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[
Id
ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342282
ExchangePattern InOnly
Headers
{breadcrumbId=ID-talend-es-edge-01-kewr0-s-missionnetworks-net-45813-1444160612354-0-21342277,
CamelNettyChannelHandlerContext=org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext@5efa3b3c,
CamelNettyLocalAddress=/69.59.252.44:11111, CamelNettyMessageEvent=[id:
0x5612eec3, /69.59.252.44:11111] RECEIVED: { "protocol":"MQDP-2",
"CQM_STREAM_INFO": "P_RTCP_XRVM_COUNT":10} ] } } from
/69.59.248.171:9999, CamelNettyRemoteAddress=/69.59.248.171:9999,
CamelRedelivered=false, CamelRedeliveryCounter=0}
BodyType String
Body { "protocol":"MQDP-2", "CQM_STREAM_INFO":
"P_RTCP_XRVM_COUNT":10} ] }}
]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
java.nio.channels.ClosedChannelException
at
org.jboss.netty.channel.socket.nio.AbstractNioWorker.cleanUpWriteBuffer(AbstractNioWorker.java:433)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.NioDatagramWorker.writeFromUserCode(NioDatagramWorker.java:212)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.NioDatagramPipelineSink.eventSunk(NioDatagramPipelineSink.java:97)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline$DefaultChannelHandlerContext.sendDownstream(DefaultChannelPipeline.java:779)
~[netty-3.9.6.Final.jar:?]
at org.jboss.netty.channel.Channels.write(Channels.java:725)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.doEncode(OneToOneEncoder.java:71)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.handler.codec.oneone.OneToOneEncoder.handleDownstream(OneToOneEncoder.java:59)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:591)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.DefaultChannelPipeline.sendDownstream(DefaultChannelPipeline.java:582)
~[netty-3.9.6.Final.jar:?]
at org.jboss.netty.channel.Channels.write(Channels.java:704)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.AbstractChannel.write(AbstractChannel.java:252)
~[netty-3.9.6.Final.jar:?]
at
org.jboss.netty.channel.socket.nio.NioDatagramChannel.write(NioDatagramChannel.java:299)
~[netty-3.9.6.Final.jar:?]
at
org.apache.camel.component.netty.NettyHelper.writeBodyAsync(NettyHelper.java:93)
[camel-netty-2.15.2.jar:2.15.2]
at
org.apache.camel.component.netty.NettyProducer.process(NettyProducer.java:263)
[camel-netty-2.15.2.jar:2.15.2]
at
org.apache.camel.processor.RedeliveryErrorHandler.process(RedeliveryErrorHandler.java:448)
[camel-core-2.15.2.jar:2.15.2]
at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:191)
[camel-core-2.15.2.jar:2.15.2]
at
org.apache.camel.util.AsyncProcessorHelper.process(AsyncProcessorHelper.java:109)
[camel-core-2.15.2.jar:2.15.2]
at
org.apache.camel.processor.DelegateAsyncProcessor.process(DelegateAsyncProcessor.java:87)
[camel-core-2.15.2.jar:2.15.2]
at
org.apache.camel.processor.WireTapProcessor$1.call(WireTapProcessor.java:119)
[camel-core-2.15.2.jar:2.15.2]
at
org.apache.camel.processor.WireTapProcessor$1.call(WireTapProcessor.java:113)
[camel-core-2.15.2.jar:2.15.2]
at java.util.concurrent.FutureTask.run(FutureTask.java:262) [?:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
[?:1.7.0_75]
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
[?:1.7.0_75]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
--
View this message in context: http://camel.465427.n5.nabble.com/camel-netty-How-to-set-the-netty-closeChannelTimeMillis-option-tp5772342p5772394.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Hi, We use an object pool (from apache common pool) to cache the open channel to save the connection time. As we don’t have the ChannelPool like the linked norbert, so we don’t have the closeChannelTimeMillis option. For you case it looks the cached the channel was closed when it is still send message. As the camel-netty always checks the channel object before borrowing it from the object pool, it could cause some trouble if you are using the UdpConnectionlessSending[1]. Can you check if you are using the UdpConnectionlessSending in your case. [1]https://github.com/apache/camel/blob/master/components/camel-netty/src/main/java/org/apache/camel/component/netty/NettyProducer.java#L571 -- Willem Jiang Blog: http://willemjiang.blogspot.com (English) http://jnn.iteye.com (Chinese) Twitter: willemjiang Weibo: 姜宁willem On October 6, 2015 at 11:44:34 PM, SteveR (srichardson@vonage.com) wrote: > On further investigation, I see that the *closeChannelTimeMillis* option is > not a *netty *option, but rather an option for > com.linkedin.norbert.javacompat.network.NetworkClientConfig > > . > > The Norbert ChannelPool closing channels prematurely > JIRA issue is still in the > unresolved state. > > We are are facing a similar issue in production (with very similar stack > traces for Norbert scenarios 2 and 3, as shown in the JIRA issue) and were > hoping for a resolution to this similar issue. > > Any thoughts on this are appreciated. > > Thanks, SteveR > > > > -- > View this message in context: http://camel.465427.n5.nabble.com/camel-netty-How-to-set-the-netty-closeChannelTimeMillis-option-tp5772342p5772347.html > Sent from the Camel - Users mailing list archive at Nabble.com. >
On further investigation, I see that the *closeChannelTimeMillis* option is not a *netty *option, but rather an option for com.linkedin.norbert.javacompat.network.NetworkClientConfig <http://grepcode.com/file/repo1.maven.org/maven2/com.linkedin/norbert_2.8.1/0.6.33/com/linkedin/norbert/javacompat/network/NetworkClientConfig.java> . The Norbert ChannelPool closing channels prematurely <https://linkedin.jira.com/browse/NOR-19> JIRA issue is still in the unresolved state. We are are facing a similar issue in production (with very similar stack traces for Norbert scenarios 2 and 3, as shown in the JIRA issue) and were hoping for a resolution to this similar issue. Any thoughts on this are appreciated. Thanks, SteveR -- View this message in context: http://camel.465427.n5.nabble.com/camel-netty-How-to-set-the-netty-closeChannelTimeMillis-option-tp5772342p5772347.html Sent from the Camel - Users mailing list archive at Nabble.com.