You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Sylvain Lebresne (JIRA)" <ji...@apache.org> on 2011/07/08 11:01:19 UTC
[jira] [Created] (CASSANDRA-2872) While dropping and recreating an
index, incremental snapshotting can hang
While dropping and recreating an index, incremental snapshotting can hang
--------------------------------------------------------------------------
Key: CASSANDRA-2872
URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.4
Reporter: Sylvain Lebresne
Priority: Minor
When creating a hard link (at list with JNA), link() hang if the target of the
link already exists. In theory though, we should not hit that situation
because we use a new directory for each manual snapshot and the generation
number of the sstables should prevent this from hapenning with increment
snapshot.
However, when you drop, then recreate a secondary index, if the sstables are
deleted after the drop and before we recreate the index, the recreated index
sstables will start with a generation to 0. Thus, when we start backuping them
incrementally, it will conflict with the sstables of the previously dropped
index.
First, we should check for the target existance because calling link() to at
least avoid hanging. But then we must make sure that when we drop, then
recreate an index, we will either not name the sstables the same way or the
incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067775#comment-13067775 ]
Jonathan Ellis commented on CASSANDRA-2872:
-------------------------------------------
Remind me why simply making sstable generation global doesn't fix this?
It's not like there's so much contention on that AtomicInteger that we need to partition it.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Priority: Minor
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067783#comment-13067783 ]
Sylvain Lebresne commented on CASSANDRA-2872:
---------------------------------------------
bq. Remind me why simply making sstable generation global doesn't fix this?
If you drop an index, then shutdown the node, then restart and recreate the index. Upon restart and crawling of the existing data files, it could be that the first available generation is the one of a sstable of the dropped index.
I guess there is two reasonably simple solutions:
# scan the (incremental) snapshot directories for the generation number too. If we do that, I guess we don't even have to make the generation global as long as we do this scanning each time a ColumnFamilyStore is created.
# make the generation number persistent in the system tables (again, no need to make the number global for that).
I think I prefer the second solution because it's more general and feels more elegant. But we would still have to scan the data dir and take the max(what we found during scan, what's in the system table) in case people force-feed data files that weren't created on that node (or the system tables are wiped).
That being said, I totally agree that the generation number don't have to be partitioned if we don't want to. But not sure it's a big deal that way either.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Priority: Minor
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2872) While dropping and recreating an
index, incremental snapshotting can hang
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2872:
--------------------------------------
Attachment: 2872.txt
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2872.txt
>
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067911#comment-13067911 ]
Hudson commented on CASSANDRA-2872:
-----------------------------------
Integrated in Cassandra-0.7 #531 (See [https://builds.apache.org/job/Cassandra-0.7/531/])
fix re-using index CF sstable names after drop/recreate
patch by jbellis; reviewed by slebresne for CASSANDRA-2872
jbellis : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1148466
Files :
* /cassandra/branches/cassandra-0.7/CHANGES.txt
* /cassandra/branches/cassandra-0.7/src/java/org/apache/cassandra/db/ColumnFamilyStore.java
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2872-v2.txt, 2872.txt
>
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2872) While dropping and recreating an
index, incremental snapshotting can hang
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2872:
--------------------------------------
Attachment: 2872-v2.txt
v2 restricts to the CF in question.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2872-v2.txt, 2872.txt
>
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064236#comment-13064236 ]
Sylvain Lebresne commented on CASSANDRA-2872:
---------------------------------------------
One option could be to have some kind of "index" generation that we would persist in the system tables. What I mean here is that if you create a index on say column 'birthdate' for some CF test, the index would start being called test.birthdate.1 (or just test.birthdate for compatibility sake), then if you drop it and recreate it, it would be called test.birthdate.2. That way, we would avoid the name clash during the incremental backup.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Priority: Minor
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067849#comment-13067849 ]
Sylvain Lebresne commented on CASSANDRA-2872:
---------------------------------------------
lgtm +1
And I can confirm that this fix the failing test of CASSANDRA-2521.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2872-v2.txt, 2872.txt
>
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13067838#comment-13067838 ]
Sylvain Lebresne commented on CASSANDRA-2872:
---------------------------------------------
bq. That's why I prefer the scan approach
Yeah, I kind of wrote my comment before as things were coming to me, in particular I wrote that I was preferring the second solution before realizing we would still need to scan. Agreeing that the scan approach is cleaner/easier.
On the patch, shouldn't we only include the sstables for the column family we're creating. I know it's ok to take the max for all cfs, but it feels a bit random unless we have only one global generation. And could be worst avoiding 2-3 mails on the mailing of people wondering why that has changed (I mean, I'm sure someone will remark it) :).
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Assignee: Jonathan Ellis
> Priority: Minor
> Fix For: 0.7.8, 0.8.2
>
> Attachments: 2872.txt
>
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2872) While dropping and recreating
an index, incremental snapshotting can hang
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13064240#comment-13064240 ]
Jonathan Ellis commented on CASSANDRA-2872:
-------------------------------------------
I really don't like messing with the name. That feels like the implementation is leaking too much into something user-visible.
> While dropping and recreating an index, incremental snapshotting can hang
> --------------------------------------------------------------------------
>
> Key: CASSANDRA-2872
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2872
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.4
> Reporter: Sylvain Lebresne
> Priority: Minor
>
> When creating a hard link (at list with JNA), link() hang if the target of the
> link already exists. In theory though, we should not hit that situation
> because we use a new directory for each manual snapshot and the generation
> number of the sstables should prevent this from hapenning with increment
> snapshot.
> However, when you drop, then recreate a secondary index, if the sstables are
> deleted after the drop and before we recreate the index, the recreated index
> sstables will start with a generation to 0. Thus, when we start backuping them
> incrementally, it will conflict with the sstables of the previously dropped
> index.
> First, we should check for the target existance because calling link() to at
> least avoid hanging. But then we must make sure that when we drop, then
> recreate an index, we will either not name the sstables the same way or the
> incremental snapshot use a different directory.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira