You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Dikang Gu (JIRA)" <ji...@apache.org> on 2011/06/03 16:13:47 UTC
[jira] [Created] (CASSANDRA-2739) Canno recover SSTable with
version f (current version g) during the node decommission.
Canno recover SSTable with version f (current version g) during the node decommission.
--------------------------------------------------------------------------------------
Key: CASSANDRA-2739
URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
Project: Cassandra
Issue Type: Bug
Components: Core
Affects Versions: 0.8.0
Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
Reporter: Dikang Gu
I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Brandon Williams (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13056873#comment-13056873 ]
Brandon Williams commented on CASSANDRA-2739:
---------------------------------------------
Note that this can be encountered for any streaming operation (such as repair) and isn't limited to decommission.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Dikang Gu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dikang Gu updated CASSANDRA-2739:
---------------------------------
Summary: Cannot recover SSTable with version f (current version g) during the node decommission. (was: Canno recover SSTable with version f (current version g) during the node decommission.)
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Sylvain Lebresne (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070633#comment-13070633 ]
Sylvain Lebresne commented on CASSANDRA-2739:
---------------------------------------------
bq. Are the steps above correct?
No. The current limitation is that when upgrading from 0.7 to 0.8, you cannot do something related to streaming (bootstrap, repair, decommission) with a mixed 0.7-0.8 cluster (more precisely, you could do some of those if the right node has the right version, but you're never guaranteed it will work) and until all sstables on the new 0.8 node have been converted to the new format.
The correct steps are:
# upgrade all node to 0.8. The all here doesn't mean all at once, it can be a rolling upgrade, but it means that streaming operation during that step may fail.
# run 'nodetool scrub' on all the node after the upgrade. The definition for all is the same than previously.
After those step, you should not get the error message anymore (otherwise, it is an unknown bug).
We will try to make it so that the second stop (the scrub part) is not necessary in the future but it is necessary for now.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058574#comment-13058574 ]
Jonathan Ellis commented on CASSANDRA-2739:
-------------------------------------------
Correct.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Mike Heffner (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13070514#comment-13070514 ]
Mike Heffner commented on CASSANDRA-2739:
-----------------------------------------
I am attempting to migrate a two-node live 0.7.6-2 cluster to 0.8.1 and I'm hitting this issue. My exact steps were:
1. Run `nodetool scrub` on both 0.7.6-2 nodes.
2. Bring one 0.8.1 node into the ring.
3. Re-run `nodetool scrub` on the 0.7.6-2 nodes and on the 0.8.1 node.
4. Run `nodetool decommission` on one of the 0.7.6-2 nodes.
Shortly after, the 0.8.1 node hits the exception above and the decommission hangs without completing. Our per-node load is about 5GB.
Are the steps above correct? Is there a way to debug this further? This is just our staging ring, but I'd like to be able to successfully live update this before doing the same with our production ring.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Ellis resolved CASSANDRA-2739.
---------------------------------------
Resolution: Duplicate
We can't provide full interoperability between upgraded and old nodes (old nodes by definition cannot handle the new-version data) but CASSANDRA-2677/CASSSANDRA-2920 do allow new nodes to stream in old-version sstables, removing the need to scrub before streaming post-upgrade.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043873#comment-13043873 ]
Stu Hood commented on CASSANDRA-2739:
-------------------------------------
Thanks for the report! As a workaround, run major compactions on all nodes before attempting to decommission again.
----
This is expected behavior, but because the f->g transition involved only the statistics component and did not change the format of the data file, this transfer should be possible with some adjustment to versioning. Perhaps a data version vs 'everything else' version, which can be derived from the a-g version?
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2739) Cannot recover
SSTable with version f (current version g) during the node decommission.
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044114#comment-13044114 ]
Stu Hood edited comment on CASSANDRA-2739 at 6/3/11 11:12 PM:
--------------------------------------------------------------
bq. I think that we may in fact be fine already since decodeKey / readRowSize / CompactedRow encapsulate the version knowledge we'd need.
The approach I've taken so far for 674 has been to move all datafile reading logic into the iterators ( see CASSANDRA-2576 and CASSANDRA-2629) which allows the index writing and sstable reading to vary by version.
What those _don't_ allow for is recognizing that the datafile for f is compatible with g, such that it can just be renamed on the destination. We might be able to warn and rewrite the file immediately on the destination if we can't just reindex it directly? This is approximately what 2576 does (but only for counters).
was (Author: stuhood):
bq. I think that we may in fact be fine already since decodeKey / readRowSize / CompactedRow encapsulate the version knowledge we'd need.
The approach I've taken so far has been to move all datafile reading logic into the iterators: see CASSANDRA-2576 and CASSANDRA-2629. What those _don't_ allow for is recognizing that the datafile for f is compatible with g, and can just be renamed on the destination.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13044114#comment-13044114 ]
Stu Hood commented on CASSANDRA-2739:
-------------------------------------
bq. I think that we may in fact be fine already since decodeKey / readRowSize / CompactedRow encapsulate the version knowledge we'd need.
The approach I've taken so far has been to move all datafile reading logic into the iterators: see CASSANDRA-2576 and CASSANDRA-2629. What those _don't_ allow for is recognizing that the datafile for f is compatible with g, and can just be renamed on the destination.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Thibaut (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13058568#comment-13058568 ]
Thibaut commented on CASSANDRA-2739:
------------------------------------
Running into the same problem after upgrading our test cluster from 0.7.* (don't know what the exact version number was) to 0.8.1. Do I have to run scrub on each node and everything will be fine afterwards?
We plan to upgrade our production cluster soon and can't afford to loose data there.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (CASSANDRA-2739) Cannot recover SSTable with
version f (current version g) during the node decommission.
Posted by "Jonathan Ellis (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043969#comment-13043969 ]
Jonathan Ellis commented on CASSANDRA-2739:
-------------------------------------------
bq. run major compactions on all nodes before attempting to decommission again
That won't necessarily fix it (since compaction will ignore single-sstable CFs), but scrub would.
Really the right fix is to commit to keeping Builder/RowIndexer backwards compatible, because this will keep coming up otherwise. I think that we may in fact be fine already since decodeKey / readRowSize / CompactedRow encapsulate the version knowledge we'd need.
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Issue Comment Edited] (CASSANDRA-2739) Cannot recover
SSTable with version f (current version g) during the node decommission.
Posted by "Stu Hood (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/CASSANDRA-2739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13043873#comment-13043873 ]
Stu Hood edited comment on CASSANDRA-2739 at 6/4/11 12:36 AM:
--------------------------------------------------------------
Thanks for the report! As a workaround, run major compactions on all nodes before attempting to decommission again.
EDIT: As jbellis said: run scrub instead.
----
This is expected behavior, but because the f->g transition involved only the statistics component and did not change the format of the data file, this transfer should be possible with some adjustment to versioning. Perhaps a data version vs 'everything else' version, which can be derived from the a-g version?
was (Author: stuhood):
Thanks for the report! As a workaround, run major compactions on all nodes before attempting to decommission again.
----
This is expected behavior, but because the f->g transition involved only the statistics component and did not change the format of the data file, this transfer should be possible with some adjustment to versioning. Perhaps a data version vs 'everything else' version, which can be derived from the a-g version?
> Cannot recover SSTable with version f (current version g) during the node decommission.
> ---------------------------------------------------------------------------------------
>
> Key: CASSANDRA-2739
> URL: https://issues.apache.org/jira/browse/CASSANDRA-2739
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 0.8.0
> Environment: centos, cassandra 0.7.4 upgrade to 0.8.0-final.
> Reporter: Dikang Gu
> Labels: decommission, version
>
> I upgrade the 4-nodes cassandra 0.7.4 cluster to 0.8.0-final. Then, I do the bin/nodetool decommission on one node, the decommission hangs there and I got the following errors on other nodes.
> ERROR [Thread-55] 2011-06-03 18:02:03,500 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-55,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
> ERROR [Thread-56] 2011-06-03 18:02:04,285 AbstractCassandraDaemon.java (line 113) Fatal exception in thread Thread[Thread-56,5,main]
> java.lang.RuntimeException: Cannot recover SSTable with version f (current version g).
> at org.apache.cassandra.io.sstable.SSTableWriter.createBuilder(SSTableWriter.java:240)
> at org.apache.cassandra.db.CompactionManager.submitSSTableBuild(CompactionManager.java:1088)
> at org.apache.cassandra.streaming.StreamInSession.finished(StreamInSession.java:108)
> at org.apache.cassandra.streaming.IncomingStreamReader.readFile(IncomingStreamReader.java:104)
> at org.apache.cassandra.streaming.IncomingStreamReader.read(IncomingStreamReader.java:61)
> at org.apache.cassandra.net.IncomingTcpConnection.stream(IncomingTcpConnection.java:155)
> at org.apache.cassandra.net.IncomingTcpConnection.run(IncomingTcpConnection.java:93)
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira