You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@activemq.apache.org by "Dominic Tootell (Created) (JIRA)" <ji...@apache.org> on 2011/10/08 19:12:29 UTC
[jira] [Created] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
-------------------------------------------------------------------------------
Key: AMQ-3530
URL: https://issues.apache.org/jira/browse/AMQ-3530
Project: ActiveMQ
Issue Type: Bug
Components: Broker
Affects Versions: 5.5.0
Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
Reporter: Dominic Tootell
When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
I have created a test case the demonstrates the above issue. The test case has the following scenario:
- 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
-- writes 10000 messages
- 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
-- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
The tests can be run as follows:
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
or
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
{noformat}
These classes are also Unit Test Cases, and output the following log files:
- Producer logs to target/producer.log
- Consumes log to target/consumer.log
- A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
- tests write to the dir *{{target/activemq-data}}*
The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
{noformat}
Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
{noformat}
To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
{noformat}
java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
----
The tests can be run as maven tests directly:
{noformat}
mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
{noformat}
----
This seems to be a different/additional issue to AMQ-2475
----
The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many
activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no
one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory
running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a
business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a
secondary concern.
Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible)
that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
a) provided a faulty patch
b) gave you a patch against a 5.4.1 version of fuse's activemq.
Big appologies for that.
thanks in advance, let me know if you need any more information.
/dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Timothy Bish (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123723#comment-13123723 ]
Timothy Bish commented on AMQ-3530:
-----------------------------------
Have you tried the trunk code? The next release will come from there so any patches would need to be applicable to that code and there's been a lot of changes since 5.5 was released.
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Description:
When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
I have created a test case the demonstrates the above issue. The test case has the following scenario:
- 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
-- writes 10000 messages
- 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
-- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
The tests can be run as follows:
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
or
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
{noformat}
These classes are also Unit Test Cases, and output the following log files:
- Producer logs to target/producer.log
- Consumes log to target/consumer.log
- A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
- tests write to the dir *{{target/activemq-data}}*
The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
{noformat}
Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
{noformat}
To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
{noformat}
java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
----
The tests can be run as maven tests directly:
{noformat}
mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
{noformat}
----
This seems to be a different/additional issue to AMQ-2475
----
The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
a) provided a faulty patch
b) gave you a patch against a 5.4.1 version of fuse's activemq.
Big appologies for that.
thanks in advance, let me know if you need any more information.
/dom
was:
When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
I have created a test case the demonstrates the above issue. The test case has the following scenario:
- 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
-- writes 10000 messages
- 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
-- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
The tests can be run as follows:
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
or
{noformat}
java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
{noformat}
These classes are also Unit Test Cases, and output the following log files:
- Producer logs to target/producer.log
- Consumes log to target/consumer.log
- A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
- tests write to the dir *{{target/activemq-data}}*
The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
{noformat}
Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
[2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
[2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
{noformat}
To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
{noformat}
java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
{noformat}
----
The tests can be run as maven tests directly:
{noformat}
mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
{noformat}
----
This seems to be a different/additional issue to AMQ-2475
----
The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many
activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no
one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory
running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a
business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a
secondary concern.
Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible)
that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
a) provided a faulty patch
b) gave you a patch against a 5.4.1 version of fuse's activemq.
Big appologies for that.
thanks in advance, let me know if you need any more information.
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123753#comment-13123753 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
Hi there,
- I checked out trunk
- build trunk
- ran the deadlock code.
I'm still seeing the deadlock occur with trunk. I'll see if I can apply the patches provided to trunk locally and run the deadlock replication again and report back.
cheers
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Patch Info: Patch Available
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128983#comment-13128983 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
I noticed that the patch to FilePendingMessageCursor as attached is irrelevant; so it can be ignored. I tested without that patch to FilePendingMessageCursor applied, and the deadlock doesn't occur (The other two patches are required in order to avoid the deadlock).
cheers,
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Attachment: deadlock.tar.gz
deadlock.tar.gz contains the mvn project that demonstrates the issue.
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many
> activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no
> one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory
> running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a
> business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a
> secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible)
> that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Labels: activemq nonPersistent storeCursor virtualTopic (was: activemq nonPersistent storeCursor)
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Attachment: StoreQueueCursor.patch.txt
FilePendingMessageCursor.patch.txt
FilePendingMessageCursor.java
Queue.patch.txt
Queue.java
StoreQueueCursor.java
Hi,
Here's my attempt at providing a patch for the issue we are encountering (this is based on the 5.5.0 source - https://svn.apache.org/repos/asf/activemq/tags/activemq-5.5.0/). I have run mvn test across the activemq 5.5.0 code base; with the patch applied; and I get the same test results as when running the test suite without the patches applied.
Please let me know if there is more information you require.
Thanks
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (AMQ-3530) Broker (5.5) Deadlock when limiting size
of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Closed) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell closed AMQ-3530.
--------------------------------
Resolution: Invalid
Hi,
Sorry, just noticed I've been a Muppet, and noticed that the problem is exactly as described in the following:
http://tmielke.blogspot.com/2011/02/observations-on-activemqs-temp-storage.html
Where the temp limit should never be set lower than the journal file size; due to the fact that the 1st temp file doesn't get removed.
The unit tests provided had exactly this issue, and didn't set the max journal size:
{noformat}
tempStore.setJournalMaxFileLength((1024*1024)*2);
{noformat}
And as a result the default 32mb, and the temp store was 10mb; resulting in the issue as described.
Upon setting the journal length to 2mb in the test; all worked as expect; no issues at all.
Apologies
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.java, StoreQueueCursor.patch.txt, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190765#comment-13190765 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
Updated StoreQueueCursor patch (StoreQueueCursor.java and StoreQueueCursor.patch.txt), so that the pendingCount is reduced if a ResourceAllocationException is thrown, otherwise it is updated; and not reduced if the error is throw.
(deadlock still occurs on trunk, tested on 5.5.1 (fuse distro 551-fuse-01-20))
thanks
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.java, StoreQueueCursor.patch.txt, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123975#comment-13123975 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
Hi,
I applied the patches to trunk locally (the same patches as detailed in the *.patch.txt(s) attached); and ran the deadlock replication code. With the patches applied I do not see the deadlock. I'm currently running the unit test suite locally, to see if there's any regression caused due to this patch (not that this is a 100% guarantee).
As the unit test suite takes a rather long time to execute; I'll report back later on.
cheers
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Labels: activemq nonPersistent storeCursor (was: )
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124975#comment-13124975 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
Hi there,
Sorry for the delay in getting back (takes a while to run the suites). I ran the unit tests for:
- vanilla trunk (no patches)
- patched trunk
The 3 tests that consistently failed in both the vanilla and patched version are:
- org.apache.activemq.bugs.AMQ2983Test
- org.apache.activemq.transport.stomp.StompNIOSSLTest
- org.apache.activemq.transport.vm.VmTransportNetworkBrokerTest
----
All tests that passed in vanilla passed in the patched version. Obviously this is not a 100% guarantee that the patches don't break anything; but some way there. I'll leave you to look them over.
Please get back to me if you have any questions
thanks
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13184178#comment-13184178 ]
Dominic Tootell commented on AMQ-3530:
--------------------------------------
Any news on this?
Thanks
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (AMQ-3530) Broker (5.5) Deadlock when limiting
size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dominic Tootell updated AMQ-3530:
---------------------------------
Attachment: StoreQueueCursor.patch.txt
StoreQueueCursor.java
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.java, StoreQueueCursor.patch.txt, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (AMQ-3530) Broker (5.5) Deadlock when
limiting size of temp store, and using VirtualTopics
Posted by "Dominic Tootell (Issue Comment Edited) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/AMQ-3530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128983#comment-13128983 ]
Dominic Tootell edited comment on AMQ-3530 at 10/18/11 9:19 AM:
----------------------------------------------------------------
I noticed that the patch to FilePendingMessageCursor as attached is irrelevant; so it can be ignored. I tested without that patch to FilePendingMessageCursor applied, and the deadlock doesn't occur (The other two patches are required in order to avoid the deadlock).
(FilePendingMessageCursor is irrelevant as the patch is for non deadlocking of the <storeCursor/> when using non persistent messages to a virtual topic, whilst restricting temp disk space. The deadlocking of <fileQueueCursor/> still occurs with the same scenario and the applied patches; and I have no found a valid solution to this. However, having one cursor free from deadlock is workable for our current use of AMQ)
cheers,
/dom
was (Author: dominict):
I noticed that the patch to FilePendingMessageCursor as attached is irrelevant; so it can be ignored. I tested without that patch to FilePendingMessageCursor applied, and the deadlock doesn't occur (The other two patches are required in order to avoid the deadlock).
cheers,
/dom
> Broker (5.5) Deadlock when limiting size of temp store, and using VirtualTopics
> -------------------------------------------------------------------------------
>
> Key: AMQ-3530
> URL: https://issues.apache.org/jira/browse/AMQ-3530
> Project: ActiveMQ
> Issue Type: Bug
> Components: Broker
> Affects Versions: 5.5.0
> Environment: 10.8.0 Darwin Kernel Version 10.8.0: Tue Jun 7 16:33:36 PDT 2011; root:xnu-1504.15.3~1/RELEASE_I386 i386
> Reporter: Dominic Tootell
> Labels: activemq, nonPersistent, storeCursor, virtualTopic
> Attachments: FilePendingMessageCursor.java, FilePendingMessageCursor.patch.txt, Queue.java, Queue.patch.txt, StoreQueueCursor.java, StoreQueueCursor.patch.txt, deadlock.tar.gz
>
>
> When using Virtual Topics and attempting to limit the Temp Table space the broker will dead lock.
> I have created a test case the demonstrates the above issue. The test case has the following scenario:
> - 1 Producer Thread (own connection) writing to VirtualTopic.FooTwo
> -- writes 10000 messages
> - 2 Consumer Threads (each own connection) consuming from Consumer.X.VirtualTopic.FooTwo
> -- consumes message (auto ack - but message ack for good measure - delay of 10ms between consumption)
> The dead lock occurs when using the either the KahaDB or ActiveMQPersistence persistence store is used, and seems more related to the KahaDB implementation used for the Temp storage area.
> I shall attach the test case (maven project), which when built (mvn clean package -DskipTests=true) will create an executable jar (target/5.5.0-deadlock-jar-with-dependencies.jar), from which the issue can be replicated:
> The tests can be run as follows:
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> or
> {noformat}
> java -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.TempStorageDeadlockReplication55
> {noformat}
> These classes are also Unit Test Cases, and output the following log files:
> - Producer logs to target/producer.log
> - Consumes log to target/consumer.log
> - A Monitor thread just run in the background that detail the number of messages sent and consumed... logs to target/monitor.log
> - tests write to the dir *{{target/activemq-data}}*
> The target/monitor.log can be tailed, upon which you can see the dead lock occur; where the consumer stops consumering and the producer stops sending; something like the following:
> {noformat}
> Dominic-Tootells-MacBook-Pro-2:trunk dominict$ tail -f monitor.log
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):0
> [2011.10.08 17:15:09] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:0
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:10] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:35
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:11] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:74
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:12] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:114
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:13] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:152
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:14] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:192
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:15] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:232
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:16] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:17] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:18] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Sent(Produced):248
> [2011.10.08 17:15:19] [Thread-14] INFO MonitorConsumerAndProducedThread - No of Message Consumed:248
> {noformat}
> To disable limiting the temp storage add the System property *{{limit.temp=false}}*; and you see that the deadlock no longer occurs:
> {noformat}
> java -Dlimit.temp=false -classpath target/5.5.0-deadlock-jar-with-dependencies.jar bbc.forge.domt.activemq.investigation.KahaDBTempStorageDeadlockReplication55
> {noformat}
> ----
> The tests can be run as maven tests directly:
> {noformat}
> mvn clean test -Dtest=KahaDBTempStorageDeadlockReplication55Test -Dlimit.temp=true
> {noformat}
> ----
> This seems to be a different/additional issue to AMQ-2475
> ----
> The reason why this issue affects our ActiveMQ environment/installation so much, is that we use a shared infrastructure model. Meaning that we provision many activemq instances on the same physcial hardware for different projects/clients. We limit the disk space assigned to each broker so that no one application goes wild and consumes all the disk space on the hardware; and as a result impacts other projects/clients. Although it theory running more than one instance of ActiveMQ on a server might seem counter intutitive (with regards to performance - disk seeking); this is a business model by which we can get the most money out of out physical hardware and provide a MOM for most projects; for which maximum throughput is a secondary concern.
> Apologies that this issue is a duplicate of AMQ-3061. However, the reason I'm opening an additional ticket is the hope that I can actually provide you a patch in here (if possible) that is coded against the 5.5 apache activemq version. As in AMQ-3061 I accidentally:
> a) provided a faulty patch
> b) gave you a patch against a 5.4.1 version of fuse's activemq.
> Big appologies for that.
> thanks in advance, let me know if you need any more information.
> /dom
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira