You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Jonathan Ellis (JIRA)" <ji...@apache.org> on 2010/11/05 22:12:45 UTC
[jira] Created: (CASSANDRA-1715) More schema migration race
conditions
More schema migration race conditions
-------------------------------------
Key: CASSANDRA-1715
URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7 beta 1
Reporter: Jonathan Ellis
Assignee: Gary Dusbabek
Priority: Critical
Fix For: 0.7.0
Related to CASSANDRA-1631.
This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
{code}
INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
...
ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
java.io.EOFException
at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
at org.apache.cassandra.db.Table.initCf(Table.java:335)
at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:619)
...
INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
{code}
There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v1-0002-compaction-lock.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: v3-0011-make-addIndex-asynchronous-and-race-proof.patch
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: v3-0012-remove-locks-from-UpdateColumnFamily.patch
new version of 06 to rebase.
11 makes addIndex asynchronous which unclogs the post-flush executor (which is single-threaded for safety, so a long index build would mean we don't clean up any commitlogs until it's done) and also makes restarting when a new index is not yet complete friendlier.
I don't think we need the locks on Update anymore since we're keeping the same Tracker and so forth so 12 removes that. locks are still needed for Drop.
What do you think?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932255#action_12932255 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
0008 addresses the flushes-within-locks brought up by jonathan. 0009 and 0010 are cleanup.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934704#action_12934704 ]
Jonathan Ellis commented on CASSANDRA-1715:
-------------------------------------------
can you open a new ticket for that?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12932052#action_12932052 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
bq. Is this saying that we'd need to tell beta3 users to rebuild their schemas if this goes in? I am fine with that, I just want to make sure I understand.
Yes, exactly.
I'll go ahead and finish this patch in the v2 direction.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0001-take-drop-off-CompactionManager.txt, v2-0002-compaction-lock.txt, v2-0003-migration-uses-locks.txt, v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0005-CFS.reload-assumes-metadata-is-mutable.txt, v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933827#action_12933827 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
bq. right, this is what 0011 does. more accurately, it uses CSLM to avoid an explicit lock.
My argument was that it wasn't safe to call indexCfs.removeAllSSTables() without the flush lock, but it looks like SSTableTracker is properly synchronized to avoid any problems with flushing. No problem here.
+1 I'll commit this shortly.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0003-migration-uses-locks.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0005-CFS.reload-assumes-metadata-is-mutable.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: v3-0012-remove-locks-from-UpdateColumnFamily.patch
new v12.
bq. the locking could conceivably be pushed down to the point in CFS when the indexes are dropped
right, this is what v11 does
bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
done.
bq. 0012 causes several unit tests to fail
one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated. fixed.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934135#action_12934135 ]
Hudson commented on CASSANDRA-1715:
-----------------------------------
Integrated in Cassandra #602 (See [https://hudson.apache.org/hudson/job/Cassandra/602/])
remove locks from UpdateColumnFamily. patch by jbellis, reviewed by gdusbabek. CASSANDRA-1715
make addIndex asynchronous and race proof. patch by jbellis, reviewed by gdusbabek. CASSANDRA-1715
remove unused fields in DropColumnFamily, DropKeyspace. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
use avro structures inside UpdateColumnFamily. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
perform index maintenance outside of migration locks during CF update. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
updateColumnFamily uses reload, remove unneccesary structures, fix bugs. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
replace modifiable CFM members with private fields and public getters. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
CFS.reload() assumes metadata is mutable. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
handle moved/dropped CF prior to pending compaction/streams. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
migration uses locks. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
compaction lock. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
take drop off CompactionManager. patch by gdusbabek, reviewe by jbellis. CASSANDRA-1715
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929893#action_12929893 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
bq. What is making files move here?
Renaming. Sorry. I know we'll need to implement it eventually, so I can't stop thinking about it. That one doesn't matter in this context.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12931890#action_12931890 ]
Jonathan Ellis commented on CASSANDRA-1715:
-------------------------------------------
The v2 approach looks great. I think the main improvement we need is to not do blocking flushes while the locks are held. For the purposes of creating a new memtable a nonblocking flush is fine. For creating indexes we'll need to set up a callback to do the index building after the flush completes. (We used to have code that took a callback arg as part of the flush call, I think I took it out but it should be relatively easy to resurrect.) I agree that it touches a lot of code, but the core changes (i.e. not one-line things like encapsulating gcgraceseconds that are messy but not dangerous) aren't much larger than v1. The huge improvement over waiting to re-sample indexes after UpdateCF is worth it imo.
I'm also fine with saying that changing the CFS will blow away any JMX-applied changes and reset values to what the new CFM says the should be. But if you are happy with the Default* approach I am too.
bq. If left for 0.7.1, I need to explain that it changes the serialization format for Migrations in a non-backwards compatible way, which is not desirable
Is this saying that we'd need to tell beta3 users to rebuild their schemas if this goes in? I am fine with that, I just want to make sure I understand.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0001-take-drop-off-CompactionManager.txt, v2-0002-compaction-lock.txt, v2-0003-migration-uses-locks.txt, v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0005-CFS.reload-assumes-metadata-is-mutable.txt, v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v1-0001-take-drop-off-CompactionManager.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt
v2-0005-CFS.reload-assumes-metadata-is-mutable.txt
v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
v2-0003-migration-uses-locks.txt
v2-0002-compaction-lock.txt
v2-0001-take-drop-off-CompactionManager.txt
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0001-take-drop-off-CompactionManager.txt, v2-0002-compaction-lock.txt, v2-0003-migration-uses-locks.txt, v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0005-CFS.reload-assumes-metadata-is-mutable.txt, v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933946#action_12933946 ]
Hudson commented on CASSANDRA-1715:
-----------------------------------
Integrated in Cassandra-0.7 #19 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/19/])
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933550#action_12933550 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
0012 causes several unit tests to fail. You might want to take a closer look at it.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930639#action_12930639 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
v2 shows what the CFS/CFM reload approach would probably look like for UpdateColumnFamily. Unfortunately it touches a lot of code and might not be warranted at this late stage in the beta cycle.
If left for 0.7.1, I need to explain that it changes the serialization format for Migrations in a non-backwards compatible way, which is not desirable. The same kind of work would have to be done for the other Migration subclasses.
This is mainly a demonstration, but one thing I'd definitely change is to use the avro CfDef in the UpdateColumnFamily constructor instead of the thrift version.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0001-take-drop-off-CompactionManager.txt, v2-0002-compaction-lock.txt, v2-0003-migration-uses-locks.txt, v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v2-0005-CFS.reload-assumes-metadata-is-mutable.txt, v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930123#action_12930123 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
bq. UpdateColumnFamily doesn't acquireLocks().
It does on line 82 in my checkout.
bq. Shouldn't Migration do that so the subclasses don't have to?
It doesn't make sense to lock on the Add* methods, but I agree it might be easier just to do the locking in the superclass. What do you think?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934706#action_12934706 ]
T Jake Luciani commented on CASSANDRA-1715:
-------------------------------------------
CASSANDRA-1768
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1715) More schema migration
race conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929815#action_12929815 ]
Jonathan Ellis edited comment on CASSANDRA-1715 at 11/8/10 6:33 PM:
--------------------------------------------------------------------
Also, out of curiosity, what were the main complications w/ mutable CFS.metadata?
was (Author: jbellis):
What was the verdict on Streaming?
Also, out of curiosity, what were the main complications w/ mutable CFS.metadata?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt
v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt
v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt
v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt
v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt
v3-0005-CFS.reload-assumes-metadata-is-mutable.txt
v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
v3-0003-migration-uses-locks.txt
v3-0002-compaction-lock.txt
v3-0001-take-drop-off-CompactionManager.txt
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0002-compaction-lock.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933544#action_12933544 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
I think update still needs to acquire the locks for the case when secondary indexes are dropped. The locking could conceivably be pushed down to the point in CFS when the indexes are dropped though, at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1715) More schema migration
race conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933812#action_12933812 ]
Jonathan Ellis edited comment on CASSANDRA-1715 at 11/19/10 9:24 AM:
---------------------------------------------------------------------
new 0012.
bq. the locking could conceivably be pushed down to the point in CFS when the indexes are dropped
right, this is what 0011 does. more accurately, it uses CSLM to avoid an explicit lock.
bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
done.
bq. 0012 causes several unit tests to fail
one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated. fixed.
was (Author: jbellis):
new 0012.
bq. the locking could conceivably be pushed down to the point in CFS when the indexes are dropped
right, this is what 0011 does
bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
done.
bq. 0012 causes several unit tests to fail
one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated. fixed.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "T Jake Luciani (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12934689#action_12934689 ]
T Jake Luciani commented on CASSANDRA-1715:
-------------------------------------------
During testing I hit this section of code:
CFMetaData.java:662
{code}
// remove the ones leaving.
for (ByteBuffer indexName : toRemove)
column_metadata.remove(indexName);
{code}
but column_metadata is defined as:
{code}
this.column_metadata = Collections.unmodifiableMap(column_metadata);
{code}
So remove() will throw an exception.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929815#action_12929815 ]
Jonathan Ellis commented on CASSANDRA-1715:
-------------------------------------------
What was the verdict on Streaming?
Also, out of curiosity, what were the main complications w/ mutable CFS.metadata?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
v1-0003-migration-uses-locks.txt
v1-0002-compaction-lock.txt
v1-0001-take-drop-off-CompactionManager.txt
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (CASSANDRA-1715) More schema migration
race conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12933812#action_12933812 ]
Jonathan Ellis edited comment on CASSANDRA-1715 at 11/19/10 9:24 AM:
---------------------------------------------------------------------
new 0012.
bq. the locking could conceivably be pushed down to the point in CFS when the indexes are dropped
right, this is what 0011 does
bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
done.
bq. 0012 causes several unit tests to fail
one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated. fixed.
was (Author: jbellis):
new v12.
bq. the locking could conceivably be pushed down to the point in CFS when the indexes are dropped
right, this is what v11 does
bq. at which point we'd need to remove the assert from the beginning of CFS.reload() and make the members that get reset in reload() volatile (minCompactionThreshold, maxCompactionThreshold, etc.).
done.
bq. 0012 causes several unit tests to fail
one was the assert, one was an unrelated commitlog bug that got exposed by something unrelated. fixed.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: (was: v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch, v3-0012-remove-locks-from-UpdateColumnFamily.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929825#action_12929825 ]
Gary Dusbabek commented on CASSANDRA-1715:
------------------------------------------
bq. what were the main complications w/ mutable CFS.metadata?
There were a couple things. A new memtable would need to know about the updated meta settings for thresholds. The timing here is tricky because of flushing (chances are you would have just flushed and have an empty memtable in anyway, but one can't be too sure). Other things... Make sure secondary indexes are dealt with properly on updates (e.g.: not reloaded needlessly). Efficiently dealing with SSTableReader instances--certain classes up updates wouldn't require messing with them at all, but others would (when files move). Ideally, it would be nice to repoint a few instances of SSTable at new data files and have all caches, stats, etc. remain intact.
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0006-replace-modifiable-CFM-members-with-private-fields-and.txt, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12929889#action_12929889 ]
Jonathan Ellis commented on CASSANDRA-1715:
-------------------------------------------
UpdateColumnFamily doesn't acquireLocks(). (Shouldn't Migration do that so the subclasses don't have to?)
bq. A new memtable would need to know about the updated meta settings for thresholds. The timing here is tricky because of flushing (chances are you would have just flushed and have an empty memtable in anyway, but one can't be too sure).
This gets a little messy code-wise (because we allow overriding memtable settings at runtime) but not too bad. At worst we just set the CFS values to the new migration values during application. I don't see any timing issues (Memtable.isThresholdViolated checks w/ the CFS each time, it doesn't cache locally).
bq. Make sure secondary indexes are dealt with properly on updates (e.g.: not reloaded needlessly).
Writing code to detect when indexes are added/dropped is a pain compared to just rebuilding it from scratch, but efficiency-wise it seems like a win. At least mutating you can avoid redoing the index sampling every time. Stopping updates in their tracks while we reload, to change read_repair_chance, is really brutal. (If UpdateCF doesn't actually need to acquireLocks then never mind, but I think it does.)
bq. Efficiently dealing with SSTableReader instances--certain classes up updates wouldn't require messing with them at all, but others would (when files move).
What is making files move here?
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v1-0001-take-drop-off-CompactionManager.txt, v1-0002-compaction-lock.txt, v1-0003-migration-uses-locks.txt, v1-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0001-take-drop-off-CompactionManager.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-1715:
--------------------------------------
Attachment: (was: v3-0012-remove-locks-from-UpdateColumnFamily.patch)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
> Attachments: v3-0001-take-drop-off-CompactionManager.txt, v3-0002-compaction-lock.txt, v3-0003-migration-uses-locks.txt, v3-0004-handle-moved-dropped-CF-prior-to-pending-compaction-st.txt, v3-0005-CFS.reload-assumes-metadata-is-mutable.txt, v3-0006-replace-modifiable-CFM-members-with-private-fields-a.patch, v3-0007-updateColumnFamily-uses-reload-remove-unneccesary-stru.txt, v3-0008-perform-index-maintenance-outside-of-migration-locks-d.txt, v3-0009-use-avro-structures-inside-UpdateColumnFamily.txt, v3-0010-remove-unused-fields-in-DropColumnFamily-DropKeyspace.txt, v3-0011-make-addIndex-asynchronous-and-race-proof.patch
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v1-0003-migration-uses-locks.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (CASSANDRA-1715) More schema migration race
conditions
Posted by "Gary Dusbabek (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Gary Dusbabek updated CASSANDRA-1715:
-------------------------------------
Attachment: (was: v2-0006-replace-modifiable-CFM-members-with-private-fields-and.txt)
> More schema migration race conditions
> -------------------------------------
>
> Key: CASSANDRA-1715
> URL: https://issues.apache.org/jira/browse/CASSANDRA-1715
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7 beta 1
> Reporter: Jonathan Ellis
> Assignee: Gary Dusbabek
> Priority: Critical
> Fix For: 0.7.0
>
>
> Related to CASSANDRA-1631.
> This is still a bug with schema updates to an existing CF, since reloadCf is doing a unload/init cycle. So flushing + compaction is an issue there as well. Here is a stacktrace from during an index creation where it stubbed its toe on an incomplete sstable from an in-progress compaction (path names anonymized):
> {code}
> INFO [CompactionExecutor:1] 2010-11-02 16:31:00,553 CompactionManager.java (line 224) Compacting [org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-6-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-7-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-8-Data.db'),org.apache.cassandra.io.sstable.SSTableReader(path='Standard1-e-9-Data.db')]
> ...
> ERROR [MigrationStage:1] 2010-11-02 16:31:10,939 ColumnFamilyStore.java (line 244) Corrupt sstable Standard1-tmp-e-10-<>=[Data.db, Index.db]; skipped
> java.io.EOFException
> at org.apache.cassandra.utils.FBUtilities.skipShortByteArray(FBUtilities.java:308)
> at org.apache.cassandra.io.sstable.SSTable.estimateRowsFromIndex(SSTable.java:231)
> at org.apache.cassandra.io.sstable.SSTableReader.load(SSTableReader.java:286)
> at org.apache.cassandra.io.sstable.SSTableReader.open(SSTableReader.java:202)
> at org.apache.cassandra.db.ColumnFamilyStore.<init>(ColumnFamilyStore.java:235)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:443)
> at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:431)
> at org.apache.cassandra.db.Table.initCf(Table.java:335)
> at org.apache.cassandra.db.Table.reloadCf(Table.java:343)
> at org.apache.cassandra.db.migration.UpdateColumnFamily.applyModels(UpdateColumnFamily.java:89)
> at org.apache.cassandra.db.migration.Migration.apply(Migration.java:158)
> at org.apache.cassandra.thrift.CassandraServer$2.call(CassandraServer.java:672)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:619)
> ...
> INFO [CompactionExecutor:1] 2010-11-02 16:31:31,970 CompactionManager.java (line 303) Compacted to Standard1-tmp-e-10-Data.db. 213,657,983 to 213,657,983 (~100% of original) bytes for 626,563 keys. Time: 31,416ms.
> {code}
> There is also a race between schema modification and streaming.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.