You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ramesh Natarajan <ra...@gmail.com> on 2011/12/16 14:46:19 UTC
gracefully recover from data file corruptions
We are running a 30 node 1.0.5 cassandra cluster running RHEL 5.6
x86_64 virtualized on ESXi 5.0. We are seeing Decorated Key assertion
error during compactions and at this point we are suspecting anything
from OS/ESXi/HBA/iSCSI RAID. Please correct me i am wrong, once a
node gets into this state I don't see any way to recover unless I
remove the corrupted data file and restart cassandra. I am running
tests with replication factor 3 and all reads and writes are done with
QUORUM. So i believe there will not be data loss if i do this.
If this is a correct way to recover I would like to know how to
gracefully do this in production environment..
- Disable thrift
- Disable gossip
- Drain the node
- kill the cassandra java process ( send a sigterm and or sigkill )
- do a filesystem sync
- remove the corrupted file from the /var/lib/cassandra/data directory
- start cassandra
- enable gossip so all pending hintedhandoff occurs
- enable thrift.
Thanks
Ramesh
Re: gracefully recover from data file corruptions
Posted by Ramesh Natarajan <ra...@gmail.com>.
Thanks Ben and Jeremiah. We are actively working with our 3rd party
vendors to determine the root cause for this issue. Hopefully we will
figure something out. This repair procedure is more like a last resort
which i really don't want to use but something to keep in mind if such
necessity arises.
thanks
Ramesh
On Fri, Dec 16, 2011 at 12:48 PM, Ben Coverston
<be...@datastax.com> wrote:
> Hi Ramesh,
>
> Every time I have seen this in the last year it has been caused by bad
> hardware or bad memory. Usually we find errors in the syslog.
>
> Jeremiah is right about running repair when you get your nodes back up.
>
> Fortunately with the addition of checksums in 1.0 I don't think that the
> corrupt data can get propagated across nodes.
>
> Your recovery steps do seem solid, if not a bit verbose. I usually tell
> people to shut down the node, remove the offending SSTables, bring the node
> back up then run repair.
>
> I can't stress enough however that if you're going to bring it back up on
> the same hardware you probably want to find the root cause, otherwise you're
> going to find yourself in the same situation days/weeks/months in the
> future.
>
> Ben
>
> On Fri, Dec 16, 2011 at 5:16 PM, Jeremiah Jordan
> <je...@morningstar.com> wrote:
>>
>> You need to run repair on the node once it is back up (to get back the
>> data you just deleted). If this is happening on more than one node you
>> could have data loss...
>>
>> -Jeremiah
>>
>>
>> On 12/16/2011 07:46 AM, Ramesh Natarajan wrote:
>>>
>>> We are running a 30 node 1.0.5 cassandra cluster running RHEL 5.6
>>> x86_64 virtualized on ESXi 5.0. We are seeing Decorated Key assertion
>>> error during compactions and at this point we are suspecting anything
>>> from OS/ESXi/HBA/iSCSI RAID. Please correct me i am wrong, once a
>>> node gets into this state I don't see any way to recover unless I
>>> remove the corrupted data file and restart cassandra. I am running
>>> tests with replication factor 3 and all reads and writes are done with
>>> QUORUM. So i believe there will not be data loss if i do this.
>>>
>>> If this is a correct way to recover I would like to know how to
>>> gracefully do this in production environment..
>>>
>>> - Disable thrift
>>> - Disable gossip
>>> - Drain the node
>>> - kill the cassandra java process ( send a sigterm and or sigkill )
>>> - do a filesystem sync
>>> - remove the corrupted file from the /var/lib/cassandra/data directory
>>> - start cassandra
>>> - enable gossip so all pending hintedhandoff occurs
>>> - enable thrift.
>>>
>>> Thanks
>>> Ramesh
>
>
>
>
> --
> Ben Coverston
> DataStax -- The Apache Cassandra Company
>
Re: gracefully recover from data file corruptions
Posted by Ben Coverston <be...@datastax.com>.
Hi Ramesh,
Every time I have seen this in the last year it has been caused by bad
hardware or bad memory. Usually we find errors in the syslog.
Jeremiah is right about running repair when you get your nodes back up.
Fortunately with the addition of checksums in 1.0 I don't think that the
corrupt data can get propagated across nodes.
Your recovery steps do seem solid, if not a bit verbose. I usually tell
people to shut down the node, remove the offending SSTables, bring the node
back up then run repair.
I can't stress enough however that if you're going to bring it back up on
the same hardware you probably want to find the root cause, otherwise
you're going to find yourself in the same situation days/weeks/months in
the future.
Ben
On Fri, Dec 16, 2011 at 5:16 PM, Jeremiah Jordan <
jeremiah.jordan@morningstar.com> wrote:
> You need to run repair on the node once it is back up (to get back the
> data you just deleted). If this is happening on more than one node you
> could have data loss...
>
> -Jeremiah
>
>
> On 12/16/2011 07:46 AM, Ramesh Natarajan wrote:
>
>> We are running a 30 node 1.0.5 cassandra cluster running RHEL 5.6
>> x86_64 virtualized on ESXi 5.0. We are seeing Decorated Key assertion
>> error during compactions and at this point we are suspecting anything
>> from OS/ESXi/HBA/iSCSI RAID. Please correct me i am wrong, once a
>> node gets into this state I don't see any way to recover unless I
>> remove the corrupted data file and restart cassandra. I am running
>> tests with replication factor 3 and all reads and writes are done with
>> QUORUM. So i believe there will not be data loss if i do this.
>>
>> If this is a correct way to recover I would like to know how to
>> gracefully do this in production environment..
>>
>> - Disable thrift
>> - Disable gossip
>> - Drain the node
>> - kill the cassandra java process ( send a sigterm and or sigkill )
>> - do a filesystem sync
>> - remove the corrupted file from the /var/lib/cassandra/data directory
>> - start cassandra
>> - enable gossip so all pending hintedhandoff occurs
>> - enable thrift.
>>
>> Thanks
>> Ramesh
>>
>
--
Ben Coverston
DataStax -- The Apache Cassandra Company
Re: gracefully recover from data file corruptions
Posted by Jeremiah Jordan <je...@morningstar.com>.
You need to run repair on the node once it is back up (to get back the
data you just deleted). If this is happening on more than one node you
could have data loss...
-Jeremiah
On 12/16/2011 07:46 AM, Ramesh Natarajan wrote:
> We are running a 30 node 1.0.5 cassandra cluster running RHEL 5.6
> x86_64 virtualized on ESXi 5.0. We are seeing Decorated Key assertion
> error during compactions and at this point we are suspecting anything
> from OS/ESXi/HBA/iSCSI RAID. Please correct me i am wrong, once a
> node gets into this state I don't see any way to recover unless I
> remove the corrupted data file and restart cassandra. I am running
> tests with replication factor 3 and all reads and writes are done with
> QUORUM. So i believe there will not be data loss if i do this.
>
> If this is a correct way to recover I would like to know how to
> gracefully do this in production environment..
>
> - Disable thrift
> - Disable gossip
> - Drain the node
> - kill the cassandra java process ( send a sigterm and or sigkill )
> - do a filesystem sync
> - remove the corrupted file from the /var/lib/cassandra/data directory
> - start cassandra
> - enable gossip so all pending hintedhandoff occurs
> - enable thrift.
>
> Thanks
> Ramesh