You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sammy Yu (JIRA)" <ji...@apache.org> on 2009/09/01 06:59:32 UTC
[jira] Issue Comment Edited: (CASSANDRA-405) Race condition with
ConcurrentLinkedHashMap
[ https://issues.apache.org/jira/browse/CASSANDRA-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749758#action_12749758 ]
Sammy Yu edited comment on CASSANDRA-405 at 8/31/09 9:59 PM:
-------------------------------------------------------------
We did multiple stack dumps they are all the same. There is no progression.
In this stack trace there are two sets of stack traces
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable [0x0000000046532000]
java.lang.Thread.State: RUNNABLE
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable [0x0000000046330000]
java.lang.Thread.State: RUNNABLE
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
All the other pool-1-threads are in the readLock waiting state due to the writeLock
"pool-1-thread-12425" prio=10 tid=0x00007f1b7857e000 nid=0x3fe6 waiting on condition [0x00007f1892528000]
java.lang.Thread.State: WAITING (parking)
at sun.misc.Unsafe.park(Native Method)
- parking to wait for 0x00007f1b8534e848> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:877)
at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1197)
at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1412)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:609)
at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:320)
at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:173)
at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:213)
at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:551)
at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:539)
at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
was (Author: sammy.yu):
In this stack trace there are two sets of threads that are stuck in the iterator
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable [0x0000000046532000]
java.lang.Thread.State: RUNNABLE
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable [0x0000000046330000]
java.lang.Thread.State: RUNNABLE
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
at org.apache.cassandra.db.Table.getRow(Table.java:589)
at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
> Race condition with ConcurrentLinkedHashMap
> -------------------------------------------
>
> Key: CASSANDRA-405
> URL: https://issues.apache.org/jira/browse/CASSANDRA-405
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.4
> Reporter: Chris Goffinet
> Attachments: stack.log.gz
>
>
> We are seeing a race condition with ConcurrentLinkedHashMap using appendToTail. We could remove the ConcurrentLinkedHashMap for now until that's resolved.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.