You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Héctor Izquierdo Seliva <iz...@strands.com> on 2011/07/06 09:55:14 UTC
Cannot recover SSTable with version f (current version g)
Hi, i've been struggling to repair my failed node for the past few days,
and I've seen this erros a few times.
java.lang.RuntimeException: Cannot recover SSTable with version f
(current version g).
If it can read the sstables, why can't they be used to repair? Is there
anything I can do besides running scrub or compact on all the cluster?
Regards
Hector Izquierdo
Re: Cannot recover SSTable with version f (current version g)
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
> That would be a bug. Can you open a ticket with the exact version
> you're using and
> the circumstance where this happens.
>
> Thanks.
>
> --
> Sylvain
https://issues.apache.org/jira/browse/CASSANDRA-2863
Re: Cannot recover SSTable with version f (current version g)
Posted by Sylvain Lebresne <sy...@datastax.com>.
2011/7/6 Héctor Izquierdo Seliva <iz...@strands.com>:
> I'm also finding a few of this while opening sstables that have been
> build with repair (SSTable build compactions)
>
> ERROR [CompactionExecutor:2] 2011-07-06 10:09:16,054
> AbstractCassandraDaemon.java (line 113) Fatal exception in thread
> Thread[CompactionExecutor:2,1,main]
> java.lang.NullPointerException
> at org.apache.cassandra.io.sstable.SSTableWriter
> $RowIndexer.close(SSTableWriter.java:382)
> at org.apache.cassandra.io.sstable.SSTableWriter
> $RowIndexer.index(SSTableWriter.java:370)
> at org.apache.cassandra.io.sstable.SSTableWriter
> $Builder.build(SSTableWriter.java:315)
> at org.apache.cassandra.db.compaction.CompactionManager
> $9.call(CompactionManager.java:1103)
> at org.apache.cassandra.db.compaction.CompactionManager
> $9.call(CompactionManager.java:1094)
> at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
> at java.util.concurrent.FutureTask.run(FutureTask.java:138)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.runTask(ThreadPoolExecutor.java:886)
> at java.util.concurrent.ThreadPoolExecutor
> $Worker.run(ThreadPoolExecutor.java:908)
> at java.lang.Thread.run(Thread.java:662)
That would be a bug. Can you open a ticket with the exact version
you're using and
the circumstance where this happens.
Thanks.
--
Sylvain
>
>
>
> El mié, 06-07-2011 a las 11:22 +0200, Sylvain Lebresne escribió:
>> 2011/7/6 Héctor Izquierdo Seliva <iz...@strands.com>:
>> > Hi, i've been struggling to repair my failed node for the past few days,
>> > and I've seen this erros a few times.
>> >
>> > java.lang.RuntimeException: Cannot recover SSTable with version f
>> > (current version g).
>> >
>> > If it can read the sstables, why can't they be used to repair?
>>
>> After a sstable is streamed (by repair in particular), we first have to recreate
>> a number of data structures. To do that, a specific part of the code is used
>> that do not know how to cope with older sstable version (hence the message).
>> This does not mean that this won't even get fixed, it will, but it is a current
>> limitation.
>>
>> > Is there anything I can do besides running scrub or compact on all the cluster?
>>
>> Not really no. You can wait enough time so that all old sstables have been
>> compacted to newer version before running a repair, but since that could take
>> a long time if you have lots of data, that may not always be an option.
>> Now probably the most "efficient" way (the quote are because it's not the most
>> efficient in time and investment it will require from you) is probably something
>> like that: For each of the nodes of the cluster:
>> 1) look the sstables on the node (do a 'ls' on the data repository).
>> The sstable
>> will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for
>> some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones.
>> 2) if all or almost all the big sstables are 'f', then run scrub on
>> that node, that'll
>> be simpler.
>> 3) if only a handfull of sstable are on 'f' (typically, if the node
>> has not just been updated),
>> then what you can do is to force a compaction on those only by using
>> JMX->CompactionManager->forceUserDefinedCompaction,
>> giving it the keyspace name as first argument and the full path to the
>> sstable as second
>> argument.
>>
>>
>>
>> >
>> > Regards
>> >
>> > Hector Izquierdo
>> >
>> >
>> >
>> >
>
>
>
Re: Cannot recover SSTable with version f (current version g)
Posted by Héctor Izquierdo Seliva <iz...@strands.com>.
Thanks Sylvain. I'll try option 3 when the current repair ends so I can
fix the remaining CFs.
I'm also finding a few of this while opening sstables that have been
build with repair (SSTable build compactions)
ERROR [CompactionExecutor:2] 2011-07-06 10:09:16,054
AbstractCassandraDaemon.java (line 113) Fatal exception in thread
Thread[CompactionExecutor:2,1,main]
java.lang.NullPointerException
at org.apache.cassandra.io.sstable.SSTableWriter
$RowIndexer.close(SSTableWriter.java:382)
at org.apache.cassandra.io.sstable.SSTableWriter
$RowIndexer.index(SSTableWriter.java:370)
at org.apache.cassandra.io.sstable.SSTableWriter
$Builder.build(SSTableWriter.java:315)
at org.apache.cassandra.db.compaction.CompactionManager
$9.call(CompactionManager.java:1103)
at org.apache.cassandra.db.compaction.CompactionManager
$9.call(CompactionManager.java:1094)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
at java.util.concurrent.FutureTask.run(FutureTask.java:138)
at java.util.concurrent.ThreadPoolExecutor
$Worker.runTask(ThreadPoolExecutor.java:886)
at java.util.concurrent.ThreadPoolExecutor
$Worker.run(ThreadPoolExecutor.java:908)
at java.lang.Thread.run(Thread.java:662)
El mié, 06-07-2011 a las 11:22 +0200, Sylvain Lebresne escribió:
> 2011/7/6 Héctor Izquierdo Seliva <iz...@strands.com>:
> > Hi, i've been struggling to repair my failed node for the past few days,
> > and I've seen this erros a few times.
> >
> > java.lang.RuntimeException: Cannot recover SSTable with version f
> > (current version g).
> >
> > If it can read the sstables, why can't they be used to repair?
>
> After a sstable is streamed (by repair in particular), we first have to recreate
> a number of data structures. To do that, a specific part of the code is used
> that do not know how to cope with older sstable version (hence the message).
> This does not mean that this won't even get fixed, it will, but it is a current
> limitation.
>
> > Is there anything I can do besides running scrub or compact on all the cluster?
>
> Not really no. You can wait enough time so that all old sstables have been
> compacted to newer version before running a repair, but since that could take
> a long time if you have lots of data, that may not always be an option.
> Now probably the most "efficient" way (the quote are because it's not the most
> efficient in time and investment it will require from you) is probably something
> like that: For each of the nodes of the cluster:
> 1) look the sstables on the node (do a 'ls' on the data repository).
> The sstable
> will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for
> some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones.
> 2) if all or almost all the big sstables are 'f', then run scrub on
> that node, that'll
> be simpler.
> 3) if only a handfull of sstable are on 'f' (typically, if the node
> has not just been updated),
> then what you can do is to force a compaction on those only by using
> JMX->CompactionManager->forceUserDefinedCompaction,
> giving it the keyspace name as first argument and the full path to the
> sstable as second
> argument.
>
>
>
> >
> > Regards
> >
> > Hector Izquierdo
> >
> >
> >
> >
Re: Cannot recover SSTable with version f (current version g)
Posted by Sylvain Lebresne <sy...@datastax.com>.
2011/7/6 Héctor Izquierdo Seliva <iz...@strands.com>:
> Hi, i've been struggling to repair my failed node for the past few days,
> and I've seen this erros a few times.
>
> java.lang.RuntimeException: Cannot recover SSTable with version f
> (current version g).
>
> If it can read the sstables, why can't they be used to repair?
After a sstable is streamed (by repair in particular), we first have to recreate
a number of data structures. To do that, a specific part of the code is used
that do not know how to cope with older sstable version (hence the message).
This does not mean that this won't even get fixed, it will, but it is a current
limitation.
> Is there anything I can do besides running scrub or compact on all the cluster?
Not really no. You can wait enough time so that all old sstables have been
compacted to newer version before running a repair, but since that could take
a long time if you have lots of data, that may not always be an option.
Now probably the most "efficient" way (the quote are because it's not the most
efficient in time and investment it will require from you) is probably something
like that: For each of the nodes of the cluster:
1) look the sstables on the node (do a 'ls' on the data repository).
The sstable
will look something like Standard1-g-105-Data.db where the 'g' may be a 'f' for
some (all?) of the sstable. The 'f' means old sstable, the 'g' means new ones.
2) if all or almost all the big sstables are 'f', then run scrub on
that node, that'll
be simpler.
3) if only a handfull of sstable are on 'f' (typically, if the node
has not just been updated),
then what you can do is to force a compaction on those only by using
JMX->CompactionManager->forceUserDefinedCompaction,
giving it the keyspace name as first argument and the full path to the
sstable as second
argument.
>
> Regards
>
> Hector Izquierdo
>
>
>
>