You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@camel.apache.org by Eric Rose <er...@athenahealth.com.INVALID> on 2021/03/15 21:43:56 UTC
direct-vm blocked in Camel 3.1 and up when using ProducerTemplate
Hello-
We are seeing an issue when we have 2 camel context communicating with each other via direct-vm calls but only when we are using the ProducerTemplate.sendBody
to send a direct-vm message back to an endpoint within a camel context that initially started the process. We are only seeing this issue after migrating from
camel-blueprint from org.apache.camel 2.17.3 to org.apache.camel.karaf 3.7.1.
After testing multiple versions of camel-blueprint it seems that the issue was introduced in version 3.1.0;
versions that are equal or below version 2.25.3 seem to work fine. We have provide some information below to show the issue when comparing version 2.25.3 and 3.1.0.
However, we are seeing the same issue with org.apache.camel.karaf 3.7.1 and only when using direct-vm.
For the given scenarios below we are using the bean "testRouting" that will send a message via ProducerTemplate within the camel-context-2
<bean id="testRotuing" class="com.ge.cedi.camelcontext2.TestRouting" />
public class TestRouting {
@Produce
protected ProducerTemplate producer;
public void sentToDirectVm() {
Map<String, String> properties = new HashMap<String, String>();
properties.put("key1", "value1");
properties.put("key2", "value2");
properties.put("key3", "value3");
producer.sendBody("direct-vm:CamelContext2_Route2", properties);
}
}
Scenario 1:
- Bundle cedi-camel-context-1 will send a message via route CamelContext1_Route1 using a direct-vm to cedi-camel-context-2 route CamelContext2_Route1.
- The message gets consumed by route CamelContext2_Route1 and will try to send a direct-vm message back to cedi-camel-context-1 route CamelContext2_Route2 using the ProducerTemplate.sendBody
bundle: cedi-camel-context-1
blueprint: camel-context-1-blueprint.xml
camelContext: CamelContext1
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<camelContext id="CamelContext1" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext1_Route1">
<from uri="scheduler://foo1?delay=40000"/>
<log message="Sending msg to CamelContext2_Route1 via blueprint direct-vm" />
<to uri="direct-vm:CamelContext1_Route1"/>
</route>
<route id="CamelContext1_Route2">
<from uri="direct-vm:CamelContext2_Route2" />
<log message="Rec Exchange from direct-vm:CamelContext2_Route2 ${body}" />
</route>
</camelContext>
</blueprint>
bundle: cedi-camel-context-2
blueprint: camel-context-2-blueprint.xml
camelContext: CamelContext2
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<bean id="testRotuing" class="com.ge.cedi.camelcontext2.TestRouting" />
<camelContext id="CamelContext2" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext2_Route1">
<from uri="direct-vm:CamelContext1_Route1"/>
<log message="Rec Exchange from direct-vm:CamelContext1_Route1 ${body}" />
<log message="Sending msg to CamelContext1_Route2 via ProducerTemplate" />
<bean ref="testRotuing" method="sentToDirectVm" />
</route>
</camelContext>
</blueprint>
- When using camel-blueprint 3.1.0 or greater; it appears that message going back to cedi-camel-context-1 does not get consumer or is being blocked.
Log file - Forcing the routes to be shutdown
2021-03-15T14:58:52,029 | WARN | FelixShutdown | DefaultAsyncProcessorAwaitManager | 55 - org.apache.camel.camel-base - 3.1.0 | Shutting down while there are still 1 inflight threads currently blocked.
2021-03-15T14:58:52,030 | WARN | FelixShutdown | DefaultAsyncProcessorAwaitManager | 55 - org.apache.camel.camel-base - 3.1.0 | The following threads are blocked and will be interrupted so the threads are released:
Blocked Thread
---------------------------------------------------------------------------------------------------------------------------------------
Id: 250
Name: Camel (CamelContext1) thread #1 - scheduler://foo1
RouteId:
NodeId:
Duration: 902873 msec.
2021-03-15T14:58:52,033 | WARN | FelixShutdown | DefaultAsyncProcessorAwaitManager | 55 - org.apache.camel.camel-base - 3.1.0 | Interrupted while waiting for asynchronous callback, will release the following blocked thread which was waiting for exchange to finish processing with exchangeId: ID-edidev02262001-1615833829150-1-2
Blocked Thread
---------------------------------------------------------------------------------------------------------------------------------------
Id: 250
Name: Camel (CamelContext1) thread #1 - scheduler://foo1
RouteId:
NodeId:
Duration: 902873 msec.
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[null ] [null ] [from[direct-vm://CamelContext2_Route2] ] [ 902876]
Exchange
---------------------------------------------------------------------------------------------------------------------------------------
Exchange[###REPLACE_ME### Id ID-edidev02262001-1615833829150-1-2###REPLACE_ME### ExchangePattern InOnly###REPLACE_ME### Headers {}###REPLACE_ME### BodyType java.util.HashMap###REPLACE_ME### Body {key1=value1, key2=value2, key3=value3}]
2021-03-15T14:58:52,044 | WARN | FelixShutdown | DefaultInflightRepository | 55 - org.apache.camel.camel-base - 3.1.0 | Shutting down while there are still 1 inflight exchanges.
2021-03-15T14:58:52,036 | ERROR | Camel (CamelContext1) thread #1 - scheduler://foo1 | DefaultErrorHandler | 63 - org.apache.camel.camel-core-engine - 3.1.0 | Failed delivery for (MessageId: ID-edidev02262001-1615833829150-0-1 on ExchangeId: ID-edidev02262001-1615833829150-0-1). Exhausted after delivery attempt: 1 caught: org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-edidev02262001-1615833829150-1-2]
Message History (complete message history is disabled)
---------------------------------------------------------------------------------------------------------------------------------------
RouteId ProcessorId Processor Elapsed (ms)
[CamelContext1_Rout] [CamelContext1_Rout] [from[scheduler://foo1?delay=40000] ] [ 902893]
...
[CamelContext1_Rout] [to1 ] [direct-vm:CamelContext1_Route1 ] [ 0]
Stacktrace
---------------------------------------------------------------------------------------------------------------------------------------
org.apache.camel.CamelExecutionException: Exception occurred during execution on the exchange: Exchange[ID-edidev02262001-1615833829150-1-2]
at org.apache.camel.CamelExecutionException.wrapCamelExecutionException(CamelExecutionException.java:47) ~[!/:3.1.0]
at org.apache.camel.support.ExchangeHelper.extractResultBody(ExchangeHelper.java:636) ~[!/:3.1.0]
at org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:568) ~[?:?]
at org.apache.camel.impl.engine.DefaultProducerTemplate.extractResultBody(DefaultProducerTemplate.java:564) ~[?:?]
at org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:189) ~[?:?]
at org.apache.camel.impl.engine.DefaultProducerTemplate.sendBody(DefaultProducerTemplate.java:195) ~[?:?]
at com.ge.cedi.camelcontext2.TestRouting.sentToDirectVm(TestRouting.java:19) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:?]
at jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) ~[?:?]
at jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[?:?]
at java.lang.reflect.Method.invoke(Method.java:566) ~[?:?]
at org.apache.camel.support.ObjectHelper.invokeMethodSafe(ObjectHelper.java:211) ~[!/:3.1.0]
at org.apache.camel.component.bean.MethodInfo.invoke(MethodInfo.java:421) ~[?:?]
at org.apache.camel.component.bean.MethodInfo$1.doProceed(MethodInfo.java:242) ~[?:?]
at org.apache.camel.component.bean.MethodInfo$1.proceed(MethodInfo.java:213) ~[?:?]
at org.apache.camel.component.bean.AbstractBeanProcessor.process(AbstractBeanProcessor.java:154) ~[?:?]
at org.apache.camel.component.bean.BeanProcessor.process(BeanProcessor.java:56) ~[?:?]
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:396) [!/:3.1.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:153) ~[!/:3.1.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60) ~[!/:3.1.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147) ~[!/:3.1.0]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:286) ~[!/:3.1.0]
at org.apache.camel.component.directvm.DirectVmProcessor.process(DirectVmProcessor.java:55) ~[?:?]
at org.apache.camel.component.directvm.DirectVmBlockingProducer.process(DirectVmBlockingProducer.java:59) ~[?:?]
at org.apache.camel.processor.SendProcessor.process(SendProcessor.java:174) ~[!/:3.1.0]
at org.apache.camel.processor.errorhandler.RedeliveryErrorHandler$SimpleTask.run(RedeliveryErrorHandler.java:396) [!/:3.1.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor$Worker.schedule(DefaultReactiveExecutor.java:153) [!/:3.1.0]
at org.apache.camel.impl.engine.DefaultReactiveExecutor.scheduleMain(DefaultReactiveExecutor.java:60) [!/:3.1.0]
at org.apache.camel.processor.Pipeline.process(Pipeline.java:147) [!/:3.1.0]
at org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:286) [!/:3.1.0]
at org.apache.camel.component.scheduler.SchedulerConsumer.sendTimerExchange(SchedulerConsumer.java:58) [!/:3.1.0]
at org.apache.camel.component.scheduler.SchedulerConsumer.poll(SchedulerConsumer.java:43) [!/:3.1.0]
at org.apache.camel.support.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:187) [!/:3.1.0]
at org.apache.camel.support.ScheduledPollConsumer.run(ScheduledPollConsumer.java:106) [!/:3.1.0]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515) [?:?]
at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:305) [?:?]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:305) [?:?]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) [?:?]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) [?:?]
at java.lang.Thread.run(Thread.java:834) [?:?]
- However, using camel-blueprint 2.25.3 or below; message process without any problems.
Scenario 2:
- Bundle cedi-camel-context-1 will send a message via route CamelContext1_Route1 using a direct-vm to cedi-camel-context-2 route CamelContext2_Route1.
- The message gets consumed by route CamelContext2_Route1 and will try to send a direct-vm message back to cedi-camel-context-1 route CamelContext2_Route2 using the blueprint direct-vm
bundle: cedi-camel-context-1
blueprint: camel-context-1-blueprint.xml
camelContext: CamelContext1
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<camelContext id="CamelContext1" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext1_Route1">
<from uri="scheduler://foo1?delay=40000"/>
<log message="Sending msg to CamelContext2_Route1 via blueprint direct-vm" />
<to uri="direct-vm:CamelContext1_Route1"/>
</route>
<route id="CamelContext1_Route2">
<from uri="direct-vm:CamelContext2_Route2" />
<log message="Rec Exchange from direct-vm:CamelContext2_Route2 ${body}" />
</route>
</camelContext>
</blueprint>
bundle: cedi-camel-context-2
blueprint: camel-context-2-blueprint.xml
camelContext: CamelContext2
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<camelContext id="CamelContext2" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext2_Route1">
<from uri="direct-vm:CamelContext1_Route1"/>
<log message="Rec Exchange from direct-vm:CamelContext1_Route1 ${body}" />
<log message="Sending msg to CamelContext1_Route2 via ProducerTemplate" />
<log message="Sending msg to CamelContext1_Route2 via blueprint direct-vm" />
<to uri="direct-vm:CamelContext2_Route2"/>
</route>
</camelContext>
</blueprint>
- Using camel-blueprint 2.17.3 or greater; message process without any problems. In this example we are using camel 3.1.0
Scenario 3:
- Bundle cedi-camel-context-1 will send a message via route CamelContext1_Route1 using a direct-vm to cedi-camel-context-2 route CamelContext2_Route1.
- The message gets consumed by route CamelContext2_Route1 and will try to send a direct-vm message to cedi-camel-context-3 route CamelContext3_Route1 using the ProducerTemplate.sendBody
bundle: cedi-camel-context-1
blueprint: camel-context-1-blueprint.xml
camelContext: CamelContext1
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<camelContext id="CamelContext1" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext1_Route1">
<from uri="scheduler://foo1?delay=40000"/>
<log message="Sending msg to CamelContext2_Route1 via blueprint direct-vm" />
<to uri="direct-vm:CamelContext1_Route1"/>
</route>
</camelContext>
</blueprint>
bundle: cedi-camel-context-2
blueprint: camel-context-2-blueprint.xml
camelContext: CamelContext2
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<bean id="testRotuing" class="com.ge.cedi.camelcontext2.TestRouting" />
<camelContext id="CamelContext2" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext2_Route1">
<from uri="direct-vm:CamelContext1_Route1"/>
<log message="Rec Exchange from direct-vm:CamelContext1_Route1 ${body}" />
<log message="Sending msg to CamelContext1_Route2 via ProducerTemplate" />
<bean ref="testRotuing" method="sentToDirectVm" />
</route>
</camelContext>
</blueprint>
bundle: cedi-camel-context-3
blueprint: camel-context-3-blueprint.xml
camelContext: CamelContext3
<?xml version="1.0" encoding="UTF-8"?>
<blueprint xmlns="http://www.osgi.org/xmlns/blueprint/v1.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://www.osgi.org/xmlns/blueprint/v1.0.0 http://www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd<http://www.osgi.org/xmlns/blueprint/v1.0.0%20http:/www.osgi.org/xmlns/blueprint/v1.0.0/blueprint.xsd>">
<camelContext id="CamelContext3" xmlns="http://camel.apache.org/schema/blueprint">
<route id="CamelContext3_Route1">
<from uri="direct-vm:CamelContext2_Route2" />
<log message="Rec Exchange from direct-vm:CamelContext2_Route2 ${body}" />
</route>
</camelContext>
</blueprint>
- Using camel-blueprint 2.17.3 or greater; message process without any problems. In this example we are using camel 3.1.0 and the ProducerTemplate.sendBody but we are sending the message from the camel-context-2 to camel-context-3.