You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Daniel Strawson (JIRA)" <ji...@apache.org> on 2012/11/19 13:20:58 UTC
[jira] [Created] (CASSANDRA-4973) Secondary Index stops returning
rows
Daniel Strawson created CASSANDRA-4973:
------------------------------------------
Summary: Secondary Index stops returning rows
Key: CASSANDRA-4973
URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 1.1.6, 1.1.2
Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
Reporter: Daniel Strawson
I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
- various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
- Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
Here is an example of the create column family I'm using for one of the CFs that affected:
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and column_metadata = [
{column_name : 'label',
validation_class : UTF8Type},
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500994#comment-13500994 ]
Daniel Strawson commented on CASSANDRA-4973:
--------------------------------------------
Hi sorry about this - the column family definition I gave in the description was wrong - I've just checked and it was off a different machine, I'd made a change to "caching" to turn on row caching. Could this be something to do with the issue? Looking through the current schema, the column families where I've noticed the index disappear all are onces where I have turned caching on. I've just checked a CF that I hadn't looked at before but has caching on and as indexes and it's giving the same issue, so whilst this isn't proof, the evidence points this way.
Here is the current output from 'show schema' for the CF 'region' :
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'ALL'
and column_metadata = [
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
{column_name : 'label',
validation_class : UTF8Type}]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13501077#comment-13501077 ]
Daniel Strawson commented on CASSANDRA-4973:
--------------------------------------------
I've tried setting caching back to 'KEYS_ONLY' on the region CF, restarted and the secondary indexes continue working, so I'm fairly happy that this issue relates to the caching setting. I will revert my other CFs back to the default setting, which is a good enough work around for me for the moment.
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Strawson updated CASSANDRA-4973:
---------------------------------------
Description:
I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
- various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
- Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
Here is an example of the create column family I'm using for one of the CFs that affected:
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and column_metadata = [
{column_name : 'label',
validation_class : UTF8Type},
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
was:
I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
- various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
- Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
Here is an example of the create column family I'm using for one of the CFs that affected:
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'KEYS_ONLY'
and column_metadata = [
{column_name : 'label',
validation_class : UTF8Type},
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500285#comment-13500285 ]
Jonathan Ellis commented on CASSANDRA-4973:
-------------------------------------------
Are there errors in your system log?
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4973) Secondary Index stops
returning rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500994#comment-13500994 ]
Daniel Strawson edited comment on CASSANDRA-4973 at 11/20/12 10:55 AM:
-----------------------------------------------------------------------
Hi sorry about this - the column family definition I gave in the description was wrong - I've just checked and it was off a different machine, I'd made a change to "caching" to turn on row caching. Could this be something to do with the issue? Looking through the current schema, the column families where I've noticed the index disappear are ones where I have turned caching on. I've just checked a CF that I hadn't looked at before but has caching on and also has secondary indexes and it's giving the same issue, so whilst this isn't proof, the evidence points towards it being something to do with caching. I will try turning off caching and see if the issue goes away.
Here is the current output from 'show schema' for the CF 'region' :
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'ALL'
and column_metadata = [
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
{column_name : 'label',
validation_class : UTF8Type}]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
was (Author: dcstrawson):
Hi sorry about this - the column family definition I gave in the description was wrong - I've just checked and it was off a different machine, I'd made a change to "caching" to turn on row caching. Could this be something to do with the issue? Looking through the current schema, the column families where I've noticed the index disappear all are onces where I have turned caching on. I've just checked a CF that I hadn't looked at before but has caching on and as indexes and it's giving the same issue, so whilst this isn't proof, the evidence points this way.
Here is the current output from 'show schema' for the CF 'region' :
create column family region
with column_type = 'Standard'
and comparator = 'UTF8Type'
and default_validation_class = 'BytesType'
and key_validation_class = 'UTF8Type'
and read_repair_chance = 0.1
and dclocal_read_repair_chance = 0.0
and gc_grace = 864000
and min_compaction_threshold = 4
and max_compaction_threshold = 32
and replicate_on_write = true
and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
and caching = 'ALL'
and column_metadata = [
{column_name : 'countryCode',
validation_class : UTF8Type,
index_name : 'region_countryCode_idx',
index_type : 0},
{column_name : 'label',
validation_class : UTF8Type}]
and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500353#comment-13500353 ]
Daniel Strawson commented on CASSANDRA-4973:
--------------------------------------------
Sorry I should have added something about that, I did get an error, but don't recall how I produced it or if it was at the point of querying the sec. index, so was hoping to put something clearer up when the issue comes back. At one point the system got into a state where queries were failing, but a restart fixed it. That and the upgrade meant that I didn't really want to put up In the mean time, this is the last exception in my system.log and I'm pretty sure would have been there when the issue was happening or arounds abouts.
ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:64)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
... 26 more
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-4973) Secondary Index stops returning
rows when caching=ALL
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Daniel Strawson updated CASSANDRA-4973:
---------------------------------------
Summary: Secondary Index stops returning rows when caching=ALL (was: Secondary Index stops returning rows)
> Secondary Index stops returning rows when caching=ALL
> -----------------------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-4973) Secondary Index stops returning
rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500405#comment-13500405 ]
Daniel Strawson commented on CASSANDRA-4973:
--------------------------------------------
ok - I've just done the following:
1. run the query "get region where countryCode=GB". This returned 13 rows OK.
2. stopped and started cassandra (kill pid).
3. ran the same query. This returned no rows.
At this point there are no errors in the log file -
Next I'm doing the following:
4. run update column family region with column_metadata=[];
5. run update column family region with column_metadata=[ ...] ; (see original schema above)
6. ran the same query, it returned 13 rows OK.
here is the log file from the point where the restart happened:
{code}
INFO [Thread-2] 2012-11-19 17:06:07,208 CassandraDaemon.java (line 212) Listening for thrift clients...
INFO [MemoryMeter:1] 2012-11-19 17:11:59,162 Memtable.java (line 213) CFS(Keyspace='Twickets', ColumnFamily='tweet') liveRatio is 4.0713678844519965 (just-counted was 4.0713678844519965). calculation took 8ms for 103 columns
INFO [MemoryMeter:1] 2012-11-19 17:11:59,164 Memtable.java (line 213) CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_737461747573_idx') liveRatio is 25.840086759341418 (just-counted was 21.256038647342994). calculation took 1ms for 9 columns
INFO [MemoryMeter:1] 2012-11-19 17:11:59,164 Memtable.java (line 213) CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_status_idx') liveRatio is 9.66183574879227 (just-counted was 9.66183574879227). calculation took 0ms for 9 columns
INFO [MemoryMeter:1] 2012-11-19 17:11:59,171 Memtable.java (line 213) CFS(Keyspace='Twickets', ColumnFamily='tweet.tweet_tweetTwitterId_idx') liveRatio is 40.57971014492754 (just-counted was 40.57971014492754). calculation took 6ms for 9 columns
INFO [RPC-Thread:95] 2012-11-19 17:13:06,280 MigrationManager.java (line 164) Update ColumnFamily 'Twickets/region' From org.apache.cassandra.config.CFMetaData@5922da6a[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=<null>,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=<null>,columnAliases=[],valueAlias=<null>,column_metadata={java.nio.HeapByteBuffer[pos=0 lim=11 cap=11]=ColumnDefinition{name=636f756e747279436f6465, validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, index_name='region_countryCode_idx'}, java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]=ColumnDefinition{name=6c6162656c, validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=null, index_name='null'}},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=<null>,caching=ALL] To org.apache.cassandra.config.CFMetaData@7f3192dd[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=<null>,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=<null>,columnAliases=[],valueAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=<null>,caching=ALL]
INFO [MigrationStage:1] 2012-11-19 17:13:06,329 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columnfamilies@216957991(1170/1462 serialized/live bytes, 20 ops)
INFO [FlushWriter:2] 2012-11-19 17:13:06,330 Memtable.java (line 264) Writing Memtable-schema_columnfamilies@216957991(1170/1462 serialized/live bytes, 20 ops)
INFO [FlushWriter:2] 2012-11-19 17:13:06,547 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-100-Data.db (1236 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=26876)
INFO [MigrationStage:1] 2012-11-19 17:13:06,548 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columns@284615223(534/667 serialized/live bytes, 10 ops)
INFO [FlushWriter:2] 2012-11-19 17:13:06,549 Memtable.java (line 264) Writing Memtable-schema_columns@284615223(534/667 serialized/live bytes, 10 ops)
INFO [CompactionExecutor:9] 2012-11-19 17:13:06,559 CompactionTask.java (line 109) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-99-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-100-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-98-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-97-Data.db')]
INFO [FlushWriter:2] 2012-11-19 17:13:06,715 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-65-Data.db (592 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=26876)
INFO [CompactionExecutor:9] 2012-11-19 17:13:06,774 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-101-Data.db,]. 42,483 to 38,815 (~91% of original) bytes for 3 keys at 0.177114MB/s. Time: 209ms.
INFO [MigrationStage:1] 2012-11-19 17:13:06,912 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-IndexInfo@759673793(48/60 serialized/live bytes, 1 ops)
INFO [FlushWriter:2] 2012-11-19 17:13:06,913 Memtable.java (line 264) Writing Memtable-IndexInfo@759673793(48/60 serialized/live bytes, 1 ops)
INFO [FlushWriter:2] 2012-11-19 17:13:07,084 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-95-Data.db (106 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=26991)
INFO [RPC-Thread:112] 2012-11-19 17:14:15,128 MigrationManager.java (line 164) Update ColumnFamily 'Twickets/region' From org.apache.cassandra.config.CFMetaData@5922da6a[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=<null>,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=<null>,columnAliases=[],valueAlias=<null>,column_metadata={},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=<null>,caching=ALL] To org.apache.cassandra.config.CFMetaData@17211aeb[cfId=1048,ksName=Twickets,cfName=region,cfType=Standard,comparator=org.apache.cassandra.db.marshal.UTF8Type,subcolumncomparator=<null>,comment=,readRepairChance=0.1,dclocalReadRepairChance=0.0,replicateOnWrite=true,gcGraceSeconds=864000,defaultValidator=org.apache.cassandra.db.marshal.BytesType,keyValidator=org.apache.cassandra.db.marshal.UTF8Type,minCompactionThreshold=4,maxCompactionThreshold=32,keyAlias=<null>,columnAliases=[],valueAlias=<null>,column_metadata={java.nio.HeapByteBuffer[pos=0 lim=11 cap=11]=ColumnDefinition{name=636f756e747279436f6465, validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=KEYS, index_name='region_countryCode_idx'}, java.nio.HeapByteBuffer[pos=0 lim=5 cap=5]=ColumnDefinition{name=6c6162656c, validator=org.apache.cassandra.db.marshal.UTF8Type, index_type=null, index_name='null'}},compactionStrategyClass=class org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy,compactionStrategyOptions={},compressionOptions={sstable_compression=org.apache.cassandra.io.compress.SnappyCompressor},bloomFilterFpChance=<null>,caching=ALL]
INFO [MigrationStage:1] 2012-11-19 17:14:15,138 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columnfamilies@2121156137(1170/1462 serialized/live bytes, 20 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,140 Memtable.java (line 264) Writing Memtable-schema_columnfamilies@2121156137(1170/1462 serialized/live bytes, 20 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,357 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columnfamilies/system-schema_columnfamilies-hf-102-Data.db (1236 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=32787)
INFO [MigrationStage:1] 2012-11-19 17:14:15,359 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-schema_columns@811753466(624/780 serialized/live bytes, 10 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,360 Memtable.java (line 264) Writing Memtable-schema_columns@811753466(624/780 serialized/live bytes, 10 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,549 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-66-Data.db (682 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=32787)
INFO [CompactionExecutor:14] 2012-11-19 17:14:15,551 CompactionTask.java (line 109) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-65-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-64-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-66-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-63-Data.db')]
INFO [MigrationStage:1] 2012-11-19 17:14:15,719 AutoSavingCache.java (line 108) reading saved cache /var/lib/cassandra/saved_caches/Twickets-region.region_countryCode_idx-KeyCache
INFO [Creating index: region.region_countryCode_idx] 2012-11-19 17:14:15,723 SecondaryIndex.java (line 168) Submitting index build of region.region_countryCode_idx for data in SSTableReader(path='/var/lib/cassandra/data/Twickets/region/Twickets-region-hd-1-Data.db')
INFO [CompactionExecutor:14] 2012-11-19 17:14:15,740 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/schema_columns/system-schema_columns-hf-67-Data.db,]. 20,308 to 16,729 (~82% of original) bytes for 2 keys at 0.084862MB/s. Time: 188ms.
INFO [MemoryMeter:1] 2012-11-19 17:14:15,789 Memtable.java (line 213) CFS(Keyspace='Twickets', ColumnFamily='region.region_countryCode_idx') liveRatio is 8.6908797417272 (just-counted was 8.6908797417272). calculation took 5ms for 65 columns
INFO [Creating index: region.region_countryCode_idx] 2012-11-19 17:14:15,812 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-region.region_countryCode_idx@885558881(1353/14698 serialized/live bytes, 71 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,813 Memtable.java (line 264) Writing Memtable-region.region_countryCode_idx@885558881(1353/14698 serialized/live bytes, 71 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,945 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/Twickets/region/Twickets-region.region_countryCode_idx-hf-1-Data.db (882 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=32787)
INFO [Creating index: region.region_countryCode_idx] 2012-11-19 17:14:15,946 ColumnFamilyStore.java (line 659) Enqueuing flush of Memtable-IndexInfo@1280290904(44/55 serialized/live bytes, 1 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:15,947 Memtable.java (line 264) Writing Memtable-IndexInfo@1280290904(44/55 serialized/live bytes, 1 ops)
INFO [FlushWriter:3] 2012-11-19 17:14:16,120 Memtable.java (line 305) Completed flushing /var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-96-Data.db (102 bytes) for commitlog position ReplayPosition(segmentId=1353344762859, position=32898)
INFO [Creating index: region.region_countryCode_idx] 2012-11-19 17:14:16,121 SecondaryIndex.java (line 209) Index build of region.region_countryCode_idx complete
INFO [CompactionExecutor:14] 2012-11-19 17:14:16,122 CompactionTask.java (line 109) Compacting [SSTableReader(path='/var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-95-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-96-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-94-Data.db'), SSTableReader(path='/var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-93-Data.db')]
INFO [CompactionExecutor:14] 2012-11-19 17:14:16,299 CompactionTask.java (line 221) Compacted to [/var/lib/cassandra/data/system/IndexInfo/system-IndexInfo-hf-97-Data.db,]. 1,727 to 1,461 (~84% of original) bytes for 1 keys at 0.007917MB/s. Time: 176ms.
{code}
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Comment Edited] (CASSANDRA-4973) Secondary Index stops
returning rows
Posted by "Daniel Strawson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-4973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500353#comment-13500353 ]
Daniel Strawson edited comment on CASSANDRA-4973 at 11/19/12 4:38 PM:
----------------------------------------------------------------------
Sorry I should have added something about that, I did get an error, but don't recall how I produced it or if it was at the point of querying the sec. index, so was hoping to put something clearer up when the issue comes back. At one point the system got into a state where queries were failing, but a restart fixed it. That and the upgrade meant that I didn't really want to put up In the mean time, this is the last exception in my system.log and I'm pretty sure would have been there when the issue was happening or arounds abouts.
{code}
ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:64)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
... 26 more
{code}
was (Author: dcstrawson):
Sorry I should have added something about that, I did get an error, but don't recall how I produced it or if it was at the point of querying the sec. index, so was hoping to put something clearer up when the issue comes back. At one point the system got into a state where queries were failing, but a restart fixed it. That and the upgrade meant that I didn't really want to put up In the mean time, this is the last exception in my system.log and I'm pretty sure would have been there when the issue was happening or arounds abouts.
ERROR [Thrift:1] 2012-11-19 11:15:41,172 AbstractCassandraDaemon.java (line 134) Exception in thread Thread[Thrift:1,5,main]java.io.IOError: java.io.EOFException
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:64)
at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1331)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1193)
at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1123)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:144)
at org.apache.cassandra.db.index.keys.KeysSearcher$1.computeNext(KeysSearcher.java:113)
at com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:140)
at com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:135)
at org.apache.cassandra.db.ColumnFamilyStore.filter(ColumnFamilyStore.java:1440)
at org.apache.cassandra.db.index.keys.KeysSearcher.search(KeysSearcher.java:88)
at org.apache.cassandra.db.index.SecondaryIndexManager.search(SecondaryIndexManager.java:595)
at org.apache.cassandra.db.ColumnFamilyStore.search(ColumnFamilyStore.java:1429)
at org.apache.cassandra.service.RangeSliceVerbHandler.executeLocally(RangeSliceVerbHandler.java:48)
at org.apache.cassandra.service.StorageProxy.getRangeSlice(StorageProxy.java:867)
at org.apache.cassandra.thrift.CassandraServer.get_indexed_slices(CassandraServer.java:811)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3056)
at org.apache.cassandra.thrift.Cassandra$Processor$get_indexed_slices.getResult(Cassandra.java:3044)
at org.apache.thrift.ProcessFunction.process(ProcessFunction.java:32)
at org.apache.thrift.TBaseProcessor.process(TBaseProcessor.java:34)
at org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:186)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
Caused by: java.io.EOFException
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:399)
at java.io.RandomAccessFile.readFully(RandomAccessFile.java:377)
at org.apache.cassandra.io.util.RandomAccessReader.readBytes(RandomAccessReader.java:324)
at org.apache.cassandra.utils.ByteBufferUtil.read(ByteBufferUtil.java:398)
at org.apache.cassandra.utils.ByteBufferUtil.readWithShortLength(ByteBufferUtil.java:380)
at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
... 26 more
> Secondary Index stops returning rows
> ------------------------------------
>
> Key: CASSANDRA-4973
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4973
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.2, 1.1.6
> Environment: Centos 6.3, Java 1.6.0_35, cass. 1.1.2 upgraded to 1.1.6
> Reporter: Daniel Strawson
>
> I've been using cassandra on a project for a little while in development and have recently suddenly started having an issue where the secondary index stops working, this is happening on my new production system, we are not yet live. Things work ok one moment, then suddenly queries to the cf through the secondary index stop returning data. I've seen it happen on 3 CFs. I've tried:
> - various nodetools repair / scrub / rebuild_indexes options, none seem to make a difference.
> - Doing a 'update column family <whatever> with column_metadata=[]' then repeating with my correct column_metadata definition. This seems to fix the problem (temporarily) until it comes back.
> The last time it happened I had just restarted cassandra, so it could be that which is causing the issue, I've got the production system ok at the moment, I will try restarting a bit later when its not being used and if I can get the issue to reoccur I will add more information.
> The problem first manifested itself in 1.1.2, so I upgraded to 1.1.6, this has not fixed it.
> Here is an example of the create column family I'm using for one of the CFs that affected:
> create column family region
> with column_type = 'Standard'
> and comparator = 'UTF8Type'
> and default_validation_class = 'BytesType'
> and key_validation_class = 'UTF8Type'
> and read_repair_chance = 0.1
> and dclocal_read_repair_chance = 0.0
> and gc_grace = 864000
> and min_compaction_threshold = 4
> and max_compaction_threshold = 32
> and replicate_on_write = true
> and compaction_strategy = 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy'
> and caching = 'KEYS_ONLY'
> and column_metadata = [
>
> {column_name : 'label',
> validation_class : UTF8Type},
>
> {column_name : 'countryCode',
> validation_class : UTF8Type,
> index_name : 'region_countryCode_idx',
> index_type : 0},
>
> ]
> and compression_options = {'sstable_compression' : 'org.apache.cassandra.io.compress.SnappyCompressor'};
> I've noticed that CASSANDRA-4785 looks similar, in my case once the system has the problem, it doesn't go away until I fix it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira