You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Suguru Namura (JIRA)" <ji...@apache.org> on 2012/08/27 06:47:07 UTC
[jira] [Created] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Suguru Namura created CASSANDRA-4578:
----------------------------------------
Summary: Dead lock in mutation stage when many concurrent writes to few columns
Key: CASSANDRA-4578
URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.3
Environment: 15 cassandra instances
CentOS5
8 Core 64GB Memory
java version "1.6.0_33"
Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
Reporter: Suguru Namura
Attachments: threaddump-1344957574788.tdump
When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
{noformat}
"MutationStage:432" - Thread t@1389
java.lang.Thread.State: TIMED_WAITING
at java.lang.Object.wait(Native Method)
- waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
at java.lang.Object.wait(Object.java:443)
at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Locked ownable synchronizers:
- locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-4578:
----------------------------------------
Fix Version/s: (was: 1.1.6)
1.1.5
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.5
>
> Attachments: 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4578) Dead lock in mutation stage
when many concurrent writes to few columns
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13449997#comment-13449997 ]
Jonathan Ellis commented on CASSANDRA-4578:
-------------------------------------------
Would prefer to have the callback final in constructor to make it more clear that it doesn't get changed during processing, otherwise +1
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.6
>
> Attachments: 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Nate McCall (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nate McCall reopened CASSANDRA-4578:
------------------------------------
Re-open for backport to 1.0.x
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.5
>
> Attachments: 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-4578:
--------------------------------------
Priority: Minor (was: Major)
Affects Version/s: (was: 1.1.3)
0.8.0
Fix Version/s: 1.1.5
Assignee: Sylvain Lebresne
You're right, since CMVH grabs a writer thread until it gets replies from the other replicas, you can have two replicas deadlock with A waiting for a reply from B, and B waiting for a reply from A.
One fix would be to move the local write into CMVH and the remote part into a separate stage (or maybe just a custom callback).
As a workaround, use CL.ONE with counters.
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.5
>
> Attachments: threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sylvain Lebresne updated CASSANDRA-4578:
----------------------------------------
Attachment: 4578.txt
Attaching patch to use a callback (as it avoids creating lots of thread that just spend time waiting on a condition) to send back the response from CMVH.
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.6
>
> Attachments: 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-4578.
---------------------------------------
Resolution: Fixed
Committed the backport to the 1.0 branch.
Not tagging with a 1.0 version since there are no plans for an official 1.0.13 release.
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.5
>
> Attachments: 4578-1.0-backport.txt, 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Nate McCall (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Nate McCall updated CASSANDRA-4578:
-----------------------------------
Attachment: 4578-1.0-backport.txt
Against cassandra-1.0 latest. Only differs in line numbers, otherwise no issues. All tests pass.
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Assignee: Sylvain Lebresne
> Priority: Minor
> Fix For: 1.1.5
>
> Attachments: 4578-1.0-backport.txt, 4578.txt, threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4578) Dead lock in mutation stage when
many concurrent writes to few columns
Posted by "Suguru Namura (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suguru Namura updated CASSANDRA-4578:
-------------------------------------
Attachment: threaddump-1344957574788.tdump
Attached thread dump
> Dead lock in mutation stage when many concurrent writes to few columns
> ----------------------------------------------------------------------
>
> Key: CASSANDRA-4578
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4578
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.3
> Environment: 15 cassandra instances
> CentOS5
> 8 Core 64GB Memory
> java version "1.6.0_33"
> Java(TM) SE Runtime Environment (build 1.6.0_33-b04)
> Java HotSpot(TM) 64-Bit Server VM (build 20.8-b03, mixed mode)
> Reporter: Suguru Namura
> Attachments: threaddump-1344957574788.tdump
>
>
> When I send many request to increment counters to few counter columns, sometimes mutation stage cause dead lock. When it happened, all of mutation threads are locked and do not accept updates any more.
> {noformat}
> "MutationStage:432" - Thread t@1389
> java.lang.Thread.State: TIMED_WAITING
> at java.lang.Object.wait(Native Method)
> - waiting on <b90b45b> (a org.apache.cassandra.utils.SimpleCondition)
> at java.lang.Object.wait(Object.java:443)
> at java.util.concurrent.TimeUnit.timedWait(TimeUnit.java:292)
> at org.apache.cassandra.utils.SimpleCondition.await(SimpleCondition.java:54)
> at org.apache.cassandra.service.AbstractWriteResponseHandler.get(AbstractWriteResponseHandler.java:55)
> at org.apache.cassandra.db.CounterMutationVerbHandler.doVerb(CounterMutationVerbHandler.java:51)
> at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:59)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> Locked ownable synchronizers:
> - locked <4b1b0a6f> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira