You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Jerru Shea (JIRA)" <ji...@apache.org> on 2008/06/25 05:23:00 UTC
[jira] Created: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Single producer and multiple consumer processes, some consumers stop getting messages.
--------------------------------------------------------------------------------------
Key: AMQ-1824
URL: https://issues.apache.org/activemq/browse/AMQ-1824
Project: ActiveMQ
Issue Type: Bug
Affects Versions: 5.1.0
Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
Reporter: Jerru Shea
Priority: Blocker
With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
{noformat}
2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
javax.transaction.xa.XAException: The connection is already closed
at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
... 19 more
2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
javax.transaction.xa.XAException: The connection is already closed
at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
at java.lang.Thread.run(Unknown Source)
Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
... 16 more
2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
{noformat}
I've written some code to reproduce this problem. Instructions:
1. unpack the attached source
2. build with 'mvn compile package'
3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46561#action_46561 ]
Jerry Shea commented on AMQ-1824:
---------------------------------
I've tried this with 5.2.0RC2 and still the same problem
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerru Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerru Shea updated AMQ-1824:
----------------------------
Attachment: repro_AMQ.zip
Reproduction test case
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerru Shea
> Priority: Blocker
> Attachments: repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Shea updated AMQ-1824:
----------------------------
Attachment: repro_AMQ_rf.zip
I don't know if the comment is quite right - apologies for not giving feedback before. Incoming messages do not use a spring JmsTemplate at all - they use xml as below as recommended on jencks website. I've uploaded a slightly refactored version of the reproduction code to make this clearer.
<bean id="inboundConnector" class="org.jencks.JCAConnector">
<property name="jcaContainer" ref="jencks" />
<property name="activationSpec" ref="activationSpec"/>
<property name="ref" value="echoBean" />
</bean>
<bean id="echoBean" class="com.repro.EchoBean" scope="prototype"/>
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Resolved: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully resolved AMQ-1824.
-----------------------------
Resolution: Working as Designed
This test is a demonstration of a limitation of the current geronimo transaction manager xid factory. With three JMVs on the same machine executing the same code, the transaction identifiers are identical.
The broker sees the same Xid on three connections instead of three separate transactions. The problem occurs when a transaction commit occurs before the last ack on a connection.
The default geronimo xidFactory impl needs to add some entropy in the form of a random to the baseId. The current use of HashCode and IP address for baseId is not sufficient. Typically nodes will be on different IP addresses so this will not be an issue.
I raised a geronimo issue with a suggested patch: https://issues.apache.org/jira/browse/GERONIMO-4393
As a workaround:
there is a constructor argument that allows the baseId to be specified, if this makes use of the clientId property, the test works as expected.
{code}
<bean id="xidFactory" class="org.apache.geronimo.transaction.manager.XidFactoryImpl">
<constructor-arg><value>bla-bla-${clientId}</value></constructor-arg>
</bean>
<bean id="transactionManager" class="org.jencks.factory.TransactionManagerFactoryBean" lazy-init="default" autowire="default" dependency-check="default" >
<property name="xidFactory" ref="xidFactory" />
</bean>
{code}
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Shea updated AMQ-1824:
----------------------------
Attachment: repro_AMQ.zip
Here's a better test case i.e. this one compiles from clean (sorry about the last one!)
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Attachments: repro_AMQ.zip, repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Filip Hanik (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=44975#action_44975 ]
Filip Hanik commented on AMQ-1824:
----------------------------------
The fact that consumer stopped receiving messages was caused by the "transaction has not been started error"
this was caused by the default JMS template in Spring would open and close the connection for each message.
Switching to the single connection factory where a single connection is reused works around this,
it is possible that using
TransactionAwareConnectionFactoryProxy will help out with this as well, when using Spring managed transactions
Filip
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Attachments: repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Bruce Snyder (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46554#action_46554 ]
Bruce Snyder commented on AMQ-1824:
-----------------------------------
I ran this up to 6500 messages/second and I'm still not seeing any issues.
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46563#action_46563 ]
Jerry Shea commented on AMQ-1824:
---------------------------------
and same problem with 5.3.SNAPSHOT I'm afraid
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46553#action_46553 ]
Gary Tully commented on AMQ-1824:
---------------------------------
I just gave your test case a run with trunk and it works as expected. Each listen gets an even distribution of messages.
I know that at least one instance of "transaction has not started" has been fixed since 5.1 as have a bunch of other high through put issues.
Can you verify that the problem still exists with trunk or with the current 5.2 RC2 - http://people.apache.org/~gtully/staging-repos/activemq-5.2.0/org/apache/activemq/apache-activemq/5.2.0 or the latest apache 5.3-SNAPSHOT build http://people.apache.org/repo/m2-snapshot-repository/org/apache/activemq/apache-activemq/5.3-SNAPSHOT/
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jerry Shea updated AMQ-1824:
----------------------------
Attachment: (was: repro_AMQ.zip)
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Attachments: repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46834#action_46834 ]
Gary Tully commented on AMQ-1824:
---------------------------------
I can now reproduce. tx not started error on broker. thx.
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Gary Tully (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Tully reassigned AMQ-1824:
-------------------------------
Assignee: Gary Tully
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Assignee: Gary Tully
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip, repro_AMQ_rf.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Jerry Shea (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=44640#action_44640 ]
Jerry Shea commented on AMQ-1824:
---------------------------------
I have managed to work around this issue by not using Jencks and instead using Spring JmsTemplate and SingleConnectionFactory
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Attachments: repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (AMQ-1824) Single producer and multiple consumer
processes, some consumers stop getting messages.
Posted by "Bruce Snyder (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/activemq/browse/AMQ-1824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=46461#action_46461 ]
Bruce Snyder commented on AMQ-1824:
-----------------------------------
Given Filip's comment, is this still an open issue?
> Single producer and multiple consumer processes, some consumers stop getting messages.
> --------------------------------------------------------------------------------------
>
> Key: AMQ-1824
> URL: https://issues.apache.org/activemq/browse/AMQ-1824
> Project: ActiveMQ
> Issue Type: Bug
> Affects Versions: 5.1.0
> Environment: Windows XP SP2, Sun JDK 1.6.03 and RHEL 64 bit Sun 1.6.05
> Reporter: Jerry Shea
> Priority: Blocker
> Fix For: 5.3.0
>
> Attachments: repro_AMQ.zip
>
>
> With a single producer and multiple consumer processes using the ActiveMQResourceAdapter (inside Jencks), some consumers stop getting messages. If you look at the JMX counters inside jconsole you can see the stuck subscription's pending queue size increasing. Nearly every time this exception is thrown by one of the failing consumers:
> {noformat}
> 2008-06-25 12:07:29,297 [pool-1-thread-7] WARN Transaction - Error ending association for XAResource org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; transaction will roll back. XA error code: -7
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:559)
> at org.apache.activemq.TransactionContext.end(TransactionContext.java:339)
> at org.apache.activemq.ra.LocalAndXATransaction.end(LocalAndXATransaction.java:90)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.end(WrapperNamedXAResource.java:51)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:563)
> at org.apache.geronimo.transaction.manager.TransactionImpl.endResources(TransactionImpl.java:542)
> at org.apache.geronimo.transaction.manager.TransactionImpl.beforePrepare(TransactionImpl.java:401)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:257)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.setXid(TransactionContext.java:556)
> ... 19 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] ERROR Transaction - Unexpected exception rolling back org.apache.geronimo.transaction.manager.WrapperNamedXAResource@17918f0; continuing with rollback
> javax.transaction.xa.XAException: The connection is already closed
> at org.apache.activemq.TransactionContext.toXAException(TransactionContext.java:624)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:423)
> at org.apache.activemq.ra.LocalAndXATransaction.rollback(LocalAndXATransaction.java:128)
> at org.apache.geronimo.transaction.manager.WrapperNamedXAResource.rollback(WrapperNamedXAResource.java:78)
> at org.apache.geronimo.transaction.manager.TransactionImpl.rollbackResources(TransactionImpl.java:581)
> at org.apache.geronimo.transaction.manager.TransactionImpl.commit(TransactionImpl.java:267)
> at org.apache.geronimo.transaction.manager.TransactionManagerImpl.commit(TransactionManagerImpl.java:238)
> at org.jencks.XAEndpoint.afterDelivery(XAEndpoint.java:105)
> at org.apache.activemq.ra.MessageEndpointProxy$MessageEndpointAlive.afterDelivery(MessageEndpointProxy.java:121)
> at org.apache.activemq.ra.MessageEndpointProxy.afterDelivery(MessageEndpointProxy.java:63)
> at org.apache.activemq.ra.ServerSessionImpl.afterDelivery(ServerSessionImpl.java:217)
> at org.apache.activemq.ActiveMQSession.run(ActiveMQSession.java:785)
> at org.apache.activemq.ra.ServerSessionImpl.run(ServerSessionImpl.java:165)
> at org.apache.geronimo.connector.work.WorkerContext.run(WorkerContext.java:290)
> at org.apache.geronimo.connector.work.pool.NamedRunnable.run(NamedRunnable.java:32)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(Unknown Source)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
> at java.lang.Thread.run(Unknown Source)
> Caused by: org.apache.activemq.ConnectionClosedException: The connection is already closed
> at org.apache.activemq.ActiveMQConnection.checkClosed(ActiveMQConnection.java:1271)
> at org.apache.activemq.ActiveMQConnection.checkClosedOrFailed(ActiveMQConnection.java:1258)
> at org.apache.activemq.TransactionContext.rollback(TransactionContext.java:407)
> ... 16 more
> 2008-06-25 12:07:29,297 [pool-1-thread-7] INFO org.apache.activemq.ra.ServerSessionImpl:0 - Endpoint failed to process message. Reason: java.lang.RuntimeException: Endpoint after delivery notification failure
> {noformat}
> I've written some code to reproduce this problem. Instructions:
> 1. unpack the attached source
> 2. build with 'mvn compile package'
> 3. start with ./startAll.sh (I used cygwin) - this will start 1 producer and 3 consumer processes. These programs are pointing to tcp://localhost:62616 (you can change this in spring config file)
> Look in jconsole under subscriptions and you will probably see within 1 or 2 runs, only 2 consumers being created, or possibly 3 consumers but 1 of them with an increasing pending queue size.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.