You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@servicemix.apache.org by "Robert H. Pollack (JIRA)" <ji...@apache.org> on 2009/04/24 21:52:38 UTC
[jira] Created: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Some Send Errors Can Cause Undeployment to Wait Forever
-------------------------------------------------------
Key: SM-1850
URL: https://issues.apache.org/activemq/browse/SM-1850
Project: ServiceMix
Issue Type: Bug
Components: servicemix-camel, servicemix-common
Affects Versions: 3.3
Reporter: Robert H. Pollack
If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
{{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
To reproduce this problem:
# Create a service assembly containing this route.
# Copy the service assembly to the {{hotdeploy}} directory.
# After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
I will attach two zip files to assist in reproducing this error:
# A pre-built service assembly that you can just copy to {{hotdeploy}}.
# Source code and a build script to assist experimentation.
(The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
The relevant code in {{EndpointDeliveryChannel}} is this:
{noformat}
public void send(MessageExchange exchange) throws MessagingException {
prepareExchange(exchange);
handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
channel.send(exchange);
}
{noformat}
The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
The relevant code in {{AsyncBaseLifeCycle}} is this:
{noformat}
public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
Set<String> exchanges = getKnownExchanges(endpoint);
synchronized (exchanges) {
if (!exchanges.isEmpty()) {
exchanges.wait();
}
}
}
{noformat}
I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
{noformat}
...
Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
id: ID:10.6.10.123-120d97c610d-3:0
status: Active
role: provider
service: {alpha:beta}gamma
in: null
] for service: {alpha:beta}gamma and interface: null
at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
... 13 more
{noformat}
Later, when attempting to shut down, a thread dump includes the following:
{noformat}
"Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x25aed9f8> (a java.util.HashSet)
at java.lang.Object.wait(Object.java:485)
at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
- locked <0x25aed9f8> (a java.util.HashSet)
at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
- locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
- locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
{noformat}
My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
{noformat}
public void send(MessageExchange exchange) throws MessagingException {
prepareExchange(exchange);
handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
try {
channel.send(exchange);
} catch (MessagingException me) {
// If something goes wrong with the send, remove this exchange
// from the list of known exchanges or else the caller will
// never be able to undeploy
// (see AsyncBaseLifeCycle.prepareShutdown()).
handleExchange (exchange, false);
throw me;
} catch (Throwable t) {
handleExchange (exchange, false);
throw new MessagingException (t.getMessage(), t);
}
}
{noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Posted by "Robert H. Pollack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51410#action_51410 ]
Robert H. Pollack commented on SM-1850:
---------------------------------------
Just for the record, and for those who may stumble across this discussion in the future, I have now tested the zip file that Gert mentions above, and it works fine. Thanks again, Gert, for your attention to this problem.
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51400#action_51400 ]
Gert Vanthienen commented on SM-1850:
-------------------------------------
Robert,
Thanks for the feedback! The shared library with the fix in it is available from the https://repository.apache.org/content/groups/snapshots/org/apache/servicemix/servicemix-shared/2009.02-SNAPSHOT/ folder. Replacing the servicemix-shared installer zip file in the hotdeploy folder with the one you find here should fix the issue.
Regards,
Gert
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (SM-1850) Some Send Errors Can Cause
Undeployment to Wait Forever
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51398#action_51398 ]
Gert Vanthienen edited comment on SM-1850 at 4/24/09 1:38 PM:
--------------------------------------------------------------
Robert,
I think this issue has already been fixed in SMXCOMP-488
Could you give it a try with a recent SNAPSHOT (available at http://repository.apache.org/snapshots) for the component and the shared library?
Thanks,
Gert
was (Author: gertvanthienen):
Robert,
I think this issue has already been in SMXCOMP-488
Could you give it a try with a recent SNAPSHOT (available at http://repository.apache.org/snapshots) for the component and the shared library?
Thanks,
Gert
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Posted by "Robert H. Pollack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert H. Pollack updated SM-1850:
----------------------------------
Attachment: ServiceMixUndeployProblem.zip
undeployTestSA.jar
The file called {{undeployTestSA.jar}} is a service assembly that can be used to reproduce this problem, as described in the report. The file called {{ServiceMixUndeployProblem.zip}} can be used to build this service assembly, in the event that there is some problem (e.g., incompatible Java versions) with using the pre-built jar.
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Posted by "Gert Vanthienen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=51398#action_51398 ]
Gert Vanthienen commented on SM-1850:
-------------------------------------
Robert,
I think this issue has already been in SMXCOMP-488
Could you give it a try with a recent SNAPSHOT (available at http://repository.apache.org/snapshots) for the component and the shared library?
Thanks,
Gert
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (SM-1850) Some Send Errors Can Cause Undeployment
to Wait Forever
Posted by "Robert H. Pollack (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/SM-1850?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert H. Pollack resolved SM-1850.
-----------------------------------
Resolution: Duplicate
I agree completely. This is indeed a duplicate of [SMXCOMP-488|https://issues.apache.org/activemq/browse/SMXCOMP-488]. I have not yet tested the snapshot distribution (I'm not yet sure how to get it), but when I look at the source code changes, they are pretty much exactly the changes I made to my own copy--which I have tested.
So I'm confident that this fix will work and I'm going to mark this issue resolved. Thank you for your prompt reply.
> Some Send Errors Can Cause Undeployment to Wait Forever
> -------------------------------------------------------
>
> Key: SM-1850
> URL: https://issues.apache.org/activemq/browse/SM-1850
> Project: ServiceMix
> Issue Type: Bug
> Components: servicemix-camel, servicemix-common
> Affects Versions: 3.3
> Reporter: Robert H. Pollack
> Attachments: ServiceMixUndeployProblem.zip, undeployTestSA.jar
>
>
> If a component causes an exception to be thrown in {{org.apache.servicemix.common.EndpointDeliveryChannel.send()}}, the exchange is nevertheless entered into a hashmap called {{knownExchanges}}. Later, when the component shuts down, the shutdown code in {{org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown()}} will wait for this exchange to be processed--but this will never happen. The result is that the JBI container completely stops.
> Reproducing this problem is easy in Camel (which is where I discovered it); it requires only a one-line Camel route such as this one:
> {{from ("timer:dmftimer1?period=60000").to ("jbi:service:alpha:beta:gamma");}}
> To reproduce this problem:
> # Create a service assembly containing this route.
> # Copy the service assembly to the {{hotdeploy}} directory.
> # After the exception is thrown, attempt to shut down the assembly (the simplest way is to control-C the ServiceMix window). The service assembly will not undeploy and the JBI container will not shut down.
> I will attach two zip files to assist in reproducing this error:
> # A pre-built service assembly that you can just copy to {{hotdeploy}}.
> # Source code and a build script to assist experimentation.
> (The build script is an Ant file that requires you to point the property {{camel-dir}} at a directory that contains the Camel core jar.
> The relevant code in {{EndpointDeliveryChannel}} is this:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> channel.send(exchange);
> }
> {noformat}
> The call to {{handleExchange()}} enters the exchange into the hashmap. But if {{channel.send()}} throws an exception, the exchange is never removed from the hashmap.
> The relevant code in {{AsyncBaseLifeCycle}} is this:
> {noformat}
> public void prepareShutdown(Endpoint endpoint) throws InterruptedException {
> Set<String> exchanges = getKnownExchanges(endpoint);
> synchronized (exchanges) {
> if (!exchanges.isEmpty()) {
> exchanges.wait();
> }
> }
> }
> {noformat}
> I do not know if all send problems cause this behavior. I discovered it by running a Camel application that attempts to send a message to a non-existent JBI endpoint. When the Camel program does this, I get a stack trace that (in part) looks like this:
> {noformat}
> ...
> Caused by: javax.jbi.messaging.MessagingException: Could not find route for exchange: InOnly[
> id: ID:10.6.10.123-120d97c610d-3:0
> status: Active
> role: provider
> service: {alpha:beta}gamma
> in: null
> ] for service: {alpha:beta}gamma and interface: null
> at org.apache.servicemix.jbi.nmr.DefaultBroker.sendExchangePacket(DefaultBroker.java:297)
> at org.apache.servicemix.jbi.security.SecuredBroker.sendExchangePacket(SecuredBroker.java:88)
> at org.apache.servicemix.jbi.container.JBIContainer.sendExchange(JBIContainer.java:894)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.doSend(DeliveryChannelImpl.java:395)
> at org.apache.servicemix.jbi.messaging.DeliveryChannelImpl.send(DeliveryChannelImpl.java:431)
> at org.apache.servicemix.common.EndpointDeliveryChannel.send(EndpointDeliveryChannel.java:88)
> at org.apache.servicemix.common.endpoints.SimpleEndpoint.send(SimpleEndpoint.java:66)
> at org.apache.servicemix.camel.CamelConsumerEndpoint.process(CamelConsumerEndpoint.java:89)
> ... 13 more
> {noformat}
> Later, when attempting to shut down, a thread dump includes the following:
> {noformat}
> "Thread-16" daemon prio=6 tid=0x2bb34400 nid=0x1398 in Object.wait() [0x2ddff000..0x2ddffc94]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x25aed9f8> (a java.util.HashSet)
> at java.lang.Object.wait(Object.java:485)
> at org.apache.servicemix.common.AsyncBaseLifeCycle.prepareShutdown(AsyncBaseLifeCycle.java:657)
> - locked <0x25aed9f8> (a java.util.HashSet)
> at org.apache.servicemix.common.DefaultServiceUnit.removeEndpoint(DefaultServiceUnit.java:205)
> - locked <0x0826a048> (a org.apache.servicemix.common.xbean.XBeanServiceUnit)
> at org.apache.servicemix.common.DefaultComponent.removeEndpoint(DefaultComponent.java:301)
> - locked <0x08269958> (a org.apache.servicemix.camel.CamelJbiComponent)
> at org.apache.servicemix.camel.JbiEndpoint$JbiProducer.stop(JbiEndpoint.java:82) ...
> {noformat}
> My proposed fix is to change the code in {{EndpointDeliveryChannel}} to a try-catch block, as shown below. But I defer to those who know ServiceMix better than I to say whether this needs to be more sophisticated. My change, which I have tested and which works fine for me, is as follows:
> {noformat}
> public void send(MessageExchange exchange) throws MessagingException {
> prepareExchange(exchange);
> handleExchange(exchange, exchange.getStatus() == ExchangeStatus.ACTIVE);
> try {
> channel.send(exchange);
> } catch (MessagingException me) {
> // If something goes wrong with the send, remove this exchange
> // from the list of known exchanges or else the caller will
> // never be able to undeploy
> // (see AsyncBaseLifeCycle.prepareShutdown()).
> handleExchange (exchange, false);
> throw me;
> } catch (Throwable t) {
> handleExchange (exchange, false);
> throw new MessagingException (t.getMessage(), t);
> }
> }
> {noformat}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.