You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jeremy Hanna (Created) (JIRA)" <ji...@apache.org> on 2011/11/13 02:07:51 UTC
[jira] [Created] (CASSANDRA-3488) Secondary indexes still around 3
months after deleting them
Secondary indexes still around 3 months after deleting them
-----------------------------------------------------------
Key: CASSANDRA-3488
URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
Project: Cassandra
Issue Type: Bug
Affects Versions: 0.8.4
Reporter: Jeremy Hanna
We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jeremy Hanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149267#comment-13149267 ]
Jeremy Hanna commented on CASSANDRA-3488:
-----------------------------------------
I also tried to update the column family information on one of the nodes where there are built indexes, thinking that might jar it into getting the right column family information. Unfortunately there are still built indexes in the column family that I tried it on.
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jeremy Hanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149213#comment-13149213 ]
Jeremy Hanna commented on CASSANDRA-3488:
-----------------------------------------
I should clarify that the built indexes only shows up on those two nodes (doing 'show keyspaces;' on the cli). On another node in the cluster, none of the built indexes appear.
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jeremy Hanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150654#comment-13150654 ]
Jeremy Hanna commented on CASSANDRA-3488:
-----------------------------------------
Did a repair on those two nodes. There aren't any built indexes on any of the column families on there any longer. For my part and our cluster, we've fixed the problem, but it sounds like there is some odd state that is possible. Maybe we close this issue but keep it as a data point if other problems show up?
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jeremy Hanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149210#comment-13149210 ]
Jeremy Hanna commented on CASSANDRA-3488:
-----------------------------------------
No. In the column family metadata for one column family, it shows no indexes, but it does show built secondary indexes:
Built indexes: [mycf.idx_id, mycf.idx_time_bucket]
It shows this on several column families.
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jonathan Ellis (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13149198#comment-13149198 ]
Jonathan Ellis commented on CASSANDRA-3488:
-------------------------------------------
Do they show in your schema?
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-3488) Secondary indexes still around
3 months after deleting them
Posted by "Jeremy Hanna (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13150042#comment-13150042 ]
Jeremy Hanna commented on CASSANDRA-3488:
-----------------------------------------
The builds of secondary indexes have continued for a third day. Talking with Brandon in IRC, he suggested I stop cassandra, move the index files to another directory, and restart those nodes. I've done that and the secondary index builds haven't come back yet. However, the last time they started after doing a repair. Once things have settled with compactions, I'm going to try another repair for each of the nodes. Things look hopeful for now that this is a workaround.
Still, sounds like there is a bug in repair or something wrt bootstrapping a node.
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-3488) Secondary indexes still around 3
months after deleting them
Posted by "Jonathan Ellis (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-3488?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-3488.
---------------------------------------
Resolution: Cannot Reproduce
I remember you had some "dropping index didn't completely drop the index" problems with this cluster dating back to 0.7.x (which sounds like CASSANDRA-2619 to me, but there were a lot of index fixes in 0.7.5 and 0.7.6). That's probably the root cause.
> Secondary indexes still around 3 months after deleting them
> -----------------------------------------------------------
>
> Key: CASSANDRA-3488
> URL: https://issues.apache.org/jira/browse/CASSANDRA-3488
> Project: Cassandra
> Issue Type: Bug
> Affects Versions: 0.8.4
> Reporter: Jeremy Hanna
> Labels: secondary_index
>
> We had an issue where we lost the data directories of two nodes. So we re-bootstrapped those two nodes into the cluster and did the repair. During the repair, secondary indexes were built as evidenced both in compactionstats and in the filesystem. The problem is, we haven't had secondary indexes on any of our column families for about 3 months now. The history there is that we were on 0.7.6, deleted the secondary indexes on that version, and upgraded to 0.8.4 a bit later. It bootstrapped and repaired last night and this morning, but it's been a good 10 hours today of almost exclusively rebuilding secondary indexes that shouldn't be there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira