You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Daniel Strawson (JIRA)" <ji...@apache.org> on 2012/11/19 17:39:58 UTC
[jira] [Comment Edited] (CASSANDRA-4973) Secondary Index stops
returning rows
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500353#comment-13500353 ]
Daniel Strawson edited comment on CASSANDRA-4973 at 11/19/12 4:38 PM:
----------------------------------------------------------------------
Sorry I should have added something about that, I did get an error, but don't recall how I produced it or if it was at the point of querying the sec. index, so was hoping to put something clearer up when the issue comes back. At one point the system got into a state where queries were failing, but a restart fixed it. That and the upgrade meant that I didn't really want to put up In the mean time, this is the last exception in my system.log and I'm pretty sure would have been there when the issue was happening or arounds abouts.
{code}
ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:64)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
... 26 more
{code}
was (Author: dcstrawson):
Sorry I should have added something about that, I did get an error, but don't recall how I produced it or if it was at the point of querying the sec. index, so was hoping to put something clearer up when the issue comes back. At one point the system got into a state where queries were failing, but a restart fixed it. That and the upgrade meant that I didn't really want to put up In the mean time, this is the last exception in my system.log and I'm pretty sure would have been there when the issue was happening or arounds abouts.
ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:64)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
... 26 more
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira