You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Keith Wright <kw...@nanigans.com> on 2013/05/07 18:41:48 UTC
mutation stalls and FileNotFoundException
I am running 1.2.4 with Vnodes and have been writing at low volume. I have doubled the volume and suddenly 3 of my 6 nodes are showing much higher load than the others (30 vs 3) and tpstats show the mutation stage as completely full (see below). I did find a FileNotFoundException that I pasted below which appears to be caused by creating, dropping, and creating a keyspace (something I did but 4 or 5 days ago). Anyone have any idea what's going on here?
Thanks
Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com
Pool Name Active Pending Completed Blocked All time blocked
ReadStage 0 0 130990 0 0
RequestResponseStage 0 0 344216 0 0
MutationStage 128 452 3464036 0 0
ReadRepairStage 0 0 14131 0 0
ReplicateOnWriteStage 0 0 32872 0 0
GossipStage 1 611 6351 0 0
AntiEntropyStage 0 0 0 0 0
MigrationStage 0 0 9 0 0
MemtablePostFlusher 0 0 91 0 0
FlushWriter 0 0 60 0 27
MiscStage 0 0 0 0 0
commitlog_archiver 0 0 0 0 0
InternalResponseStage 0 0 3 0 0
HintedHandoff 1 1 13 0 0
Message type Dropped
RANGE_SLICE 0
READ_REPAIR 54
BINARY 0
READ 0
MUTATION 8539
_TRACE 0
REQUEST_RESPONSE 0
ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555 CassandraDaemon.java (line 174) Exception in thread Thread[ReplicateOnWriteStage:95404,5,main]
java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:722)
Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
at org.apache.cassandra.db.Table.getRow(Table.java:347)
at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
... 3 more
Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
at java.io.RandomAccessFile.open(Native Method)
at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
... 21 more
Re: mutation stalls and FileNotFoundException
Posted by aaron morton <aa...@thelastpickle.com>.
When mutation stage messages are dropped I start by looking for GC problems in the cassandra logs.
You also have some blocked flush writer tasks, which may be due to a large number of CF's, a large number of secondary indexes, slow disk IO or excessive use of snapshot / flush. Check the comments for the memtable_flush_queue_size in the yaml file.
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Consultant
New Zealand
@aaronmorton
http://www.thelastpickle.com
On 8/05/2013, at 4:41 AM, Keith Wright <kw...@nanigans.com> wrote:
> I am running 1.2.4 with Vnodes and have been writing at low volume. I have doubled the volume and suddenly 3 of my 6 nodes are showing much higher load than the others (30 vs 3) and tpstats show the mutation stage as completely full (see below). I did find a FileNotFoundException that I pasted below which appears to be caused by creating, dropping, and creating a keyspace (something I did but 4 or 5 days ago). Anyone have any idea what's going on here?
>
> Thanks
>
> Keiths-MacBook-Pro:bin keith$ ./nodetool tpstats -h lxpcas005.nanigans.com
> Pool Name Active Pending Completed Blocked All time blocked
> ReadStage 0 0 130990 0 0
> RequestResponseStage 0 0 344216 0 0
> MutationStage 128 452 3464036 0 0
> ReadRepairStage 0 0 14131 0 0
> ReplicateOnWriteStage 0 0 32872 0 0
> GossipStage 1 611 6351 0 0
> AntiEntropyStage 0 0 0 0 0
> MigrationStage 0 0 9 0 0
> MemtablePostFlusher 0 0 91 0 0
> FlushWriter 0 0 60 0 27
> MiscStage 0 0 0 0 0
> commitlog_archiver 0 0 0 0 0
> InternalResponseStage 0 0 3 0 0
> HintedHandoff 1 1 13 0 0
>
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 54
> BINARY 0
> READ 0
> MUTATION 8539
> _TRACE 0
> REQUEST_RESPONSE 0
>
>
>
> ERROR [ReplicateOnWriteStage:95404] 2013-05-06 14:55:06,555 CassandraDaemon.java (line 174) Exception in thread Thread[ReplicateOnWriteStage:95404,5,main]
> java.lang.RuntimeException: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1582)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:722)
> Caused by: java.lang.RuntimeException: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
> at org.apache.cassandra.io.util.CompressedSegmentedFile.createReader(CompressedSegmentedFile.java:57)
> at org.apache.cassandra.io.util.PoolingSegmentedFile.getSegment(PoolingSegmentedFile.java:41)
> at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:976)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.createFileDataInput(SSTableNamesIterator.java:98)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.read(SSTableNamesIterator.java:117)
> at org.apache.cassandra.db.columniterator.SSTableNamesIterator.<init>(SSTableNamesIterator.java:64)
> at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:81)
> at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:68)
> at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:274)
> at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:65)
> at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1357)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1214)
> at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1126)
> at org.apache.cassandra.db.Table.getRow(Table.java:347)
> at org.apache.cassandra.db.SliceByNamesReadCommand.getRow(SliceByNamesReadCommand.java:64)
> at org.apache.cassandra.db.CounterMutation.makeReplicationMutation(CounterMutation.java:90)
> at org.apache.cassandra.service.StorageProxy$7$1.runMayThrow(StorageProxy.java:796)
> at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1578)
> ... 3 more
> Caused by: java.io.FileNotFoundException: /data/1/cassandra/data/users/global_user_stats/users-global_user_stats-ib-30716-Data.db (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.<init>(RandomAccessFile.java:233)
> at org.apache.cassandra.io.util.RandomAccessReader.<init>(RandomAccessReader.java:67)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:75)
> at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:42)
> ... 21 more