You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Erik Onnen (JIRA)" <ji...@apache.org> on 2011/03/07 22:00:59 UTC
[jira] Created: (CASSANDRA-2283) Streaming Old Format Data Fails in
0.7.3 after upgrade from 0.6.8
Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
-----------------------------------------------------------------
Key: CASSANDRA-2283
URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.7.3
Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
Reporter: Erik Onnen
After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.EOFException
at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.EOFException
at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
java.io.EOFException
at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13012077#comment-13012077 ]
Jonathan Ellis commented on CASSANDRA-2283:
-------------------------------------------
You're right, making it one-pass isn't feasible without writing the streamed row out to a temporary file first, since we don't have a way to rebuild the row-level bloom filter + block index. In other words, not really any more one-pass than the existing approach.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011597#comment-13011597 ]
Stu Hood commented on CASSANDRA-2283:
-------------------------------------
> (a) rewrite it to latest format and (b) write bloom filter and row index
I was hoping that we could get away with just doing (b) in order to avoid having to re-write the data file, but it certainly simplifies things to re-write in the current format.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13006090#comment-13006090 ]
Stu Hood commented on CASSANDRA-2283:
-------------------------------------
While not ideal, this is actually supposed to throw an exception in SSTableWriter.createBuilder, but that is dependent on a correctly versioned Descriptor being created on the destination side. I expect that streaming is dropping the source version when it creates the destination descriptor.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13021315#comment-13021315 ]
Hudson commented on CASSANDRA-2283:
-----------------------------------
Integrated in Cassandra-0.7 #442 (See [https://hudson.apache.org/hudson/job/Cassandra-0.7/442/])
preserve version when streaming data from old sstables
patch by jbellis; reviewed by Stu Hood for CASSANDRA-2283
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.1
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
> Assignee: Jonathan Ellis
> Fix For: 0.7.5
>
> Attachments: 2283.txt
>
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004580#comment-13004580 ]
Jonathan Ellis commented on CASSANDRA-2283:
-------------------------------------------
CASSANDRA-2296 is going to cause streaming trouble too. In other words: scrub won't fix this unless run w/ a build that has 2296 applied.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13004658#comment-13004658 ]
Jonathan Ellis commented on CASSANDRA-2283:
-------------------------------------------
bq. scrub won't fix this
that is, for sstables containing expired tombstones
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Erik Onnen (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13010916#comment-13010916 ]
Erik Onnen commented on CASSANDRA-2283:
---------------------------------------
Ok, I can confirm that after upgrading to 0.7.4 where 2296 was applied, and after performing a scrub, we were able to bootstrap in new nodes again.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13019085#comment-13019085 ]
Stu Hood commented on CASSANDRA-2283:
-------------------------------------
Looks good: only comment is that BootstrapTest should probably purposely use an old version and check that it is preserved.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.1
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
> Assignee: Jonathan Ellis
> Fix For: 0.7.5
>
> Attachments: 2283.txt
>
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis updated CASSANDRA-2283:
--------------------------------------
Attachment: 2283.txt
Patch to preserve version across streams. Also removes obsolete component field from PendingFile (we only stream Component.DATA).
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.1
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
> Fix For: 0.7.5
>
> Attachments: 2283.txt
>
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2283) Streaming Old Format Data Fails
in 0.7.3 after upgrade from 0.6.8
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011360#comment-13011360 ]
Jonathan Ellis commented on CASSANDRA-2283:
-------------------------------------------
IMO the right thing to do here is to deserialize enough of the data sent during stream to (a) rewrite it to latest format and (b) write bloom filter and row index -- currently this is done in a second pass post-stream.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2283) Streaming Old Format
Data Fails in 0.7.3 after upgrade from 0.6.8
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2283?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13011597#comment-13011597 ]
Stu Hood edited comment on CASSANDRA-2283 at 3/26/11 9:53 AM:
--------------------------------------------------------------
> (a) rewrite it to latest format and (b) write bloom filter and row index
I was hoping that we could get away with just doing (b) in order to avoid having to re-write the data file, but it certainly simplifies things to re-write in the current format.
EDIT: CASSANDRA-2336 took a step toward allowing rebuilding and index writing to be version specific, in order to implement (b). I'm most of the way through an implementation of CASSANDRA-2319 on top of it, but I don't see a clear answer for a/b/a+b.
was (Author: stuhood):
> (a) rewrite it to latest format and (b) write bloom filter and row index
I was hoping that we could get away with just doing (b) in order to avoid having to re-write the data file, but it certainly simplifies things to re-write in the current format.
> Streaming Old Format Data Fails in 0.7.3 after upgrade from 0.6.8
> -----------------------------------------------------------------
>
> Key: CASSANDRA-2283
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2283
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.7.3
> Environment: 0.7.3 upgraded from 0.6.8, Linux Java HotSpot(TM) 64-Bit Server VM (build 17.1-b03, mixed mode)
> Reporter: Erik Onnen
>
> After successfully upgrading a 0.6.8 ring to 0.7.3, we needed to bootstrap in a new node relatively quickly. When starting the new node with an assigned token in auto bootstrap mode, we see the following exceptions on the new node:
> INFO [main] 2011-03-07 10:37:32,671 StorageService.java (line 505) Joining: sleeping 30000 ms for pending range setup
> INFO [main] 2011-03-07 10:38:02,679 StorageService.java (line 505) Bootstrapping
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,899 HintedHandOffManager.java (line 304) Started hinted handoff for endpoint /10.211.14.200
> INFO [HintedHandoff:1] 2011-03-07 10:38:02,900 HintedHandOffManager.java (line 360) Finished hinted handoff of 0 rows to endpoint /10.211.14.200
> INFO [CompactionExecutor:1] 2011-03-07 10:38:04,924 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,390 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuff-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:05,768 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-1
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,389 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-2
> INFO [CompactionExecutor:1] 2011-03-07 10:38:06,581 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-3
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:07,056 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,480 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid-f-5
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,582 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-1
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,635 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> ERROR [CompactionExecutor:1] 2011-03-07 10:38:08,666 AbstractCassandraDaemon.java (line 114) Fatal exception in thread Thread[CompactionExecutor:1,1,main]
> java.io.EOFException
> at org.apache.cassandra.io.sstable.IndexHelper.skipIndex(IndexHelper.java:65)
> at org.apache.cassandra.io.sstable.SSTableWriter$Builder.build(SSTableWriter.java:303)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:923)
> at org.apache.cassandra.db.CompactionManager$9.call(CompactionManager.java:916)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
> INFO [CompactionExecutor:1] 2011-03-07 10:38:08,855 SSTableReader.java (line 154) Opening /mnt/services/cassandra/var/data/0.7.3/data/Stuff/stuffid_reg_idx-f-4
> Two attempts to bootstrap in the new node both exhibited this behavior. On the node owning the tokens being migrated, stream activity is visible but doesn't update any progress I think due to the issues on the receiving host.
> Lastly, just case it's relevant, we had an EC2 node die underneath us during the upgrade so not all nodes were drained. This didn't affect the upgrade but I wanted to note it her to be thorough.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira