You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by SteveR <sr...@vonage.com> on 2015/10/26 19:35:14 UTC
NEED HELP: Exchanges remain inflight on route with recipientList to
netty:udp
Hi:
*What I'm using:* Linux RHEL6, Camel 2.15.3, netty3, Java 1.7
I have to deliver my Camel application to production soon and I have a big
problem that I can't figure out.
I have a route that consumes from a *SEDA *queue, formulates a UDP
acknowledgement from the received exchange body, and then sends the ack back
to the associated UDP client specified in the
*NettyConstants.NETTY_REMOTE_ADDRESS* header on the exchange.
The UDP acks do get sent out, which I have verified via tcpdump, but when I
look at the Camel JMX route statistics (using VisualVM) it shows that all
exchanges flowing through this ack-back route are marked as
*ExchangesInflight* and remain that way. If I run a long test, say sending
in a million messages, the application experiences
*java.net.SocketException: Too many open files.*
I'm including my route below. By the way, I'm using the route-scoped
*onCompletion().onCompleteOnly()* approach within another route to send to
this ack-back route.
Any help on how to fix this is desperately needed and greatly appreciated!!!
I was also hoping that adding .stop()
<http://camel.apache.org/maven/camel-2.15.0/camel-core/apidocs/org/apache/camel/model/ProcessorDefinition.html#stop%28%29>
to the route would force it to mark each exchange as completed, but it
doesn't appear to.
Also, the only reason I'm using the *recipientList* EIP is because of the
suggestion on the How do I use dynamic URI
<http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html> webpage. So
far, it's the only way I could figure out how to dynamically apply the
remote host and remote port to the netty URI. I only need to send each ACK
to a single UDP client host, but I don't know of any other way to make it
work dynamically. You would think that if there is a
*NettyConstants.NETTY_REMOTE_ADDRESS* header on the exchange that there
would be a way to tell *camel-netty* to just use that remote address.
Thanks, SteveR
from(uri="seda:SEDA_ACK_QUEUE?size=200000&concurrentConsumers=10&waitForTaskToComplete=Never&failIfNoConsumers=true&timeout=1000")
.setExchangePattern(ExchangePattern.InOnly)
.routeId(ackQueueRouteId)
.startupOrder(ackQueueRouteStartupOrder)
.setProperty(Exchange.CHARSET_NAME,
ExpressionBuilder.constantExpression(charsetName))
.threads(threadPoolSize /*poolSize*/, threadPoolSize * 2 /*maxPoolSize*/)
.threadName("threads_" + sourceRouteId + "_ACK_QUEUE")
.process(cqmsAckBackProcessor).id(cqmsAckBackProcessorId) // Formulate
ACK
.recipientList( // See
http://camel.apache.org/how-do-i-use-dynamic-uri-in-to.html
simple(
"netty:udp://${header.REMOTE_HOST_IP}:" +
"${header.REMOTE_PORT_NUMBER}?" +
"clientPipelineFactory=#MY_CLIENT_PIPELINE_FACTORY&" +
"sendBufferSize=26214400&allowDefaultCodec=false"
)
)
.setExchangePattern(ExchangePattern.InOnly)
.stop() // Stops subsequent routing of the current Exchange and marks it
as completed.
.end(); // Ends the current block.
--
View this message in context: http://camel.465427.n5.nabble.com/NEED-HELP-Exchanges-remain-inflight-on-route-with-recipientList-to-netty-udp-tp5773079.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: NEED HELP: Exchanges remain inflight on route with
recipientList() or toD()
Posted by SteveR <sr...@vonage.com>.
By the way, on the ack-back route in question that sends the UDP
acknowledgement via *netty:udp*, I've tried with and without the
*udpConnectionlessSending *option and neither way works.
--
View this message in context: http://camel.465427.n5.nabble.com/NEED-HELP-Exchanges-remain-inflight-on-route-with-recipientList-or-toD-tp5773079p5773082.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: NEED HELP: Exchanges remain inflight on route with
recipientList to netty:udp
Posted by SteveR <sr...@vonage.com>.
Ok, I just tried upgrading to *Camel 2.16.0* in the hope that the new
Dynamic To <http://camel.apache.org/message-endpoint.html> would solve
this problem, but I still get the same result, that is all exchanges that
are sent to the route stay marked as *ExchangesInflight*. This time I even
had a UDP listener running using the Linux socat
<http://www.dest-unreach.org/socat/doc/socat.html> command and it receives
the correct number of UDP acknowledgements. So the route is sending all
exchanges that it received on its SEDA queue, but they never transition to
the completed state.
Below is my updated route that uses the new *toD()* method. Below that is
the route which sends to this ack-back route.
I'm really puzzled by this behavior, but I need to figure it out soon
because my management is getting cranky ;-(
Thanks for any help, SteveR
from(uri="seda:SEDA_ACK_QUEUE?size=200000&concurrentConsumers=10")
.setExchangePattern(ExchangePattern.InOnly)
.routeId(ackQueueRouteId)
.startupOrder(ackQueueRouteStartupOrder)
.setProperty(Exchange.CHARSET_NAME,
ExpressionBuilder.constantExpression(charsetName))
.process(cqmsAckBackProcessor).id(cqmsAckBackProcessorId)
* .toD(
"netty:udp://${header.REMOTE_HOST_IP}:" +
"${header.REMOTE_PORT_NUMBER}?" +
"clientPipelineFactory=#MY_CLIENT_PIPELINE_FACTORY&" +
"sendBufferSize=26214400&allowDefaultCodec=false"
)*
.stop().end();
from(kafkaQueueURI)
.errorHandler(deadLetterChannelBuilder) // Add route-scoped
DeadLetterChannel error handler
* .onCompletion()
.parallelProcessing() // Tells Camel to use a thread pool for
onCompletion route
.onCompleteOnly() // Synchronize only after Exchange completes
successfully with no errors
.to(ackQueueURI).id(sourceRouteId + "_ON_COMPLETION_ONLY_TO_ACK_QUEUE")
.end() // Must use end() to denote the end of the onCompletion route*
.setExchangePattern(ExchangePattern.InOnly)
.routeId(kafkaQueueRouteId)
.startupOrder(kafkaQueueRouteStartupOrder)
.setProperty(Exchange.CHARSET_NAME,
ExpressionBuilder.constantExpression(charsetName))
.threads(threadPoolSize /*poolSize*/, threadPoolSize * 2 /*maxPoolSize*/)
.threadName("threads_" + kafkaQueueRouteId)
.callerRunsWhenRejected(true) // Hard-coded since we always want this
behavior!
//
----------------------------------------------------------------------------------------
// This processor handles Kafka related processing, e.g. determining the
Kafka partitioning
//
----------------------------------------------------------------------------------------
.process(kafkaProcessor).id(kafkaProcessorId)
// ---------------------------------------------------
// Here we route to the final destination (e.g. Kafka)
// ---------------------------------------------------
.to(kafkaToURI)
.setProperty(Exchange.CHARSET_NAME,
ExpressionBuilder.constantExpression(charsetName));
<http://camel.465427.n5.nabble.com/file/n5773080/IN_FLIGHT_1.png>
--
View this message in context: http://camel.465427.n5.nabble.com/NEED-HELP-Exchanges-remain-inflight-on-route-with-recipientList-to-netty-udp-tp5773079p5773080.html
Sent from the Camel - Users mailing list archive at Nabble.com.
Re: NEED HELP: Exchanges remain inflight on route with
recipientList() or toD()
Posted by SteveR <sr...@vonage.com>.
This problem is severe enough to cause my Camel application to exhaust
resources (e.g. number of open files) which results in major application
failure. Below are some of the errors and stack trace that I get when
sending in only 4000 messages using a Java UDP sender test program.
I've worked pretty hard, so far, trying to learn Camel and standing up this
UDP routing application. Everything else seems to be working just fine (e.g.
UDP receive, producing to kafka, creating backup files, throughput logging,
etc). This route that deals with sending a UDP acknowledgement is the only
part that is causing issues. I would hate to give up now over an issue like
this, but I'm running out of ideas and my management is getting impatient
and a jaded view of Apache Camel. It was my idea to use Camel, so I'm
trying real hard to make it work successfully.
Thanks, SteveR
[2015-10-26 21:08:05,451 UTC] *ERROR *[Camel (MPLR_CQMS) thread #40 -
seda://SEDA_ACK_103] [FailureEventNotifier:notify:63]: Exchange failed:
exchangeId =
ID-talend-es-01-dloco-m-mission-net-32956-1445893522490-0-51974, fromRouteId
= ROUTE_ID_RAW_CQMS_EVENTS_SEDA_ACK, route =
EventDrivenConsumerRoute[Endpoint[seda://SEDA_ACK_103?concurrentConsumers=10&size=200000]
-> Pipeline[[Channel[SetExchangePattern[InOnly]],
Channel[SetProperty(CamelCharsetName, iso-8859-1)],
Channel[DelegateSync[com.mission.mplr.multiprotocollistenerrouter.CqmsAckBackProcessor@3563739]],
Channel[sendTo(netty:udp://${header.REMOTE_HOST_IP}:${header.REMOTE_PORT_NUMBER}?clientPipelineFactory=#CQMS_CLIENT_PIPELINE_FACTORY_ROUTE_ID_RAW_CQMS_EVENTS&sendBufferSize=26214400&allowDefaultCodec=false)],
Channel[Stop]]]], *exceptionMsg = Failed to open a DatagramChannel.*
[2015-10-26 21:09:37,663] *WARN
*[qtp2066624561-39-acceptor-0@1b7d21a5-ServerConnector@5dcdfc98{HTTP/1.1}{talend-es-01.dloco.m.mission.net:8090}]
*java.io.IOException: Too many open files*
at sun.nio.ch.ServerSocketChannelImpl.accept0(Native Method)
~[?:1.7.0_75]
at
sun.nio.ch.ServerSocketChannelImpl.accept(ServerSocketChannelImpl.java:241)
~[?:1.7.0_75]
at
org.eclipse.jetty.server.ServerConnector.accept(ServerConnector.java:377)
~[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.server.AbstractConnector$Acceptor.run(AbstractConnector.java:500)
[jetty-server-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at
org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
[jetty-util-9.2.11.v20150529.jar:9.2.11.v20150529]
at java.lang.Thread.run(Thread.java:745) [?:1.7.0_75]
--
View this message in context: http://camel.465427.n5.nabble.com/NEED-HELP-Exchanges-remain-inflight-on-route-with-recipientList-or-toD-tp5773079p5773081.html
Sent from the Camel - Users mailing list archive at Nabble.com.