You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rene Kochen <re...@schange.com> on 2012/09/07 15:38:01 UTC
Node-tool drain on Cassandra 1.0
Hi All,
I have a question about node-tool drain on a single Cassandra 1.0.11 node:
If I use node-tool drain, it does stop accepting writes and flushes the
tables. However, is it normal that the commit log files are not deleted and
that it gets replayed?
Because if I do the following:
1) Write some records.
2) Drain the node.
3) At this point, the commit log file still contains data.
3) Stop Cassandra service.
4) Delete all data files (not the commit log).
5) Start the Cassandra service
It replays the commit log and the records are created again. Why doesn't it
skip the commit log?
In the log it says:
discard completed log segments for ReplayPosition(segmentId=1347024392876,
position=170545), column family 1011.
Not safe to delete commit log
CommitLogSegment(D:\cassandradatabase\Log\CommitLog-1347024392876.log);
dirty is Versions (7), ; hasNext: false
Re: Node-tool drain on Cassandra 1.0
Posted by Rob Coli <rc...@palominodb.com>.
On Sun, Sep 9, 2012 at 12:01 PM, Robin Verlangen <ro...@us2.nl> wrote:
> Deleting the commitlog files is harmless. It's just a tool that tries to
> keep Cassandra more in-sync with the other nodes. A standard repair will fix
> all problems that a commitlog replay might do too.
This is not really true.. imagine a RF=2 cluster.
1) take a replica node down
2) write at CL.ONE to another replica node
3) replication fails to other replica due to it being down, hint is
queued locally, this means both writes are only in memtables/mirrored
in the commitlog
4) don't nodetool flush
5) stop node
6) delete commitlog
You have now lost data, and repair can't fix it, because the data
you've lost has not been written to any other node. This is one of the
edge cases that makes CL.ONE pretty risky if you care about your data
and use a RF under 3.
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb
Re: Node-tool drain on Cassandra 1.0
Posted by Robin Verlangen <ro...@us2.nl>.
Deleting the commitlog files is harmless. It's just a tool that tries to
keep Cassandra more in-sync with the other nodes. A standard repair will
fix all problems that a commitlog replay might do too.
Best regards,
Robin Verlangen
*Software engineer*
*
*
W http://www.robinverlangen.nl
E robin@us2.nl
Disclaimer: The information contained in this message and attachments is
intended solely for the attention and use of the named addressee and may be
confidential. If you are not the intended recipient, you are reminded that
the information remains the property of the sender. You must not use,
disclose, distribute, copy, print or rely on this e-mail. If you have
received this message in error, please contact the sender immediately and
irrevocably delete this message and any copies.
2012/9/8 Rene Kochen <re...@schange.com>
> OK, thanks! I will vote for that ticket.
>
> On a production system, I have an extremely big table. I want to
> physically delete it. It it safe to just delete the commit log files after
> a drain?
>
> 1) Drain node
> 2) Stop Cassandra
> 3) Delete commit log files
> 4) Delete all files related to the big table
> 5) Restart Cassandra
>
> Thanks
>
> Rene
>
>
> 2012/9/7 Rob Coli <rc...@palominodb.com>
>
>> On Fri, Sep 7, 2012 at 6:38 AM, Rene Kochen <re...@schange.com>
>> wrote:
>> > If I use node-tool drain, it does stop accepting writes and flushes the
>> > tables. However, is it normal that the commit log files are not deleted
>> and
>> > that it gets replayed?
>>
>> It's not expected by design, but it does seem to be normal in
>> cassandra 1.0.x. I've spoken with other operators and they anecdotally
>> report the same behavior when doing the same operation you describe.
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-4446
>>
>> The more people who report that they have the issue, the greater
>> chance of a response or fix, so I suggest commenting "me too!" on that
>> ticket.. :)
>>
>> =Rob
>>
>> --
>> =Robert Coli
>> AIM>ALK - rcoli@palominodb.com
>> YAHOO - rcoli.palominob
>> SKYPE - rcoli_palominodb
>>
>
>
Re: Node-tool drain on Cassandra 1.0
Posted by Rene Kochen <re...@schange.com>.
OK, thanks! I will vote for that ticket.
On a production system, I have an extremely big table. I want to physically
delete it. It it safe to just delete the commit log files after a drain?
1) Drain node
2) Stop Cassandra
3) Delete commit log files
4) Delete all files related to the big table
5) Restart Cassandra
Thanks
Rene
2012/9/7 Rob Coli <rc...@palominodb.com>
> On Fri, Sep 7, 2012 at 6:38 AM, Rene Kochen <re...@schange.com>
> wrote:
> > If I use node-tool drain, it does stop accepting writes and flushes the
> > tables. However, is it normal that the commit log files are not deleted
> and
> > that it gets replayed?
>
> It's not expected by design, but it does seem to be normal in
> cassandra 1.0.x. I've spoken with other operators and they anecdotally
> report the same behavior when doing the same operation you describe.
>
> https://issues.apache.org/jira/browse/CASSANDRA-4446
>
> The more people who report that they have the issue, the greater
> chance of a response or fix, so I suggest commenting "me too!" on that
> ticket.. :)
>
> =Rob
>
> --
> =Robert Coli
> AIM>ALK - rcoli@palominodb.com
> YAHOO - rcoli.palominob
> SKYPE - rcoli_palominodb
>
Re: Node-tool drain on Cassandra 1.0
Posted by Rob Coli <rc...@palominodb.com>.
On Fri, Sep 7, 2012 at 6:38 AM, Rene Kochen <re...@schange.com> wrote:
> If I use node-tool drain, it does stop accepting writes and flushes the
> tables. However, is it normal that the commit log files are not deleted and
> that it gets replayed?
It's not expected by design, but it does seem to be normal in
cassandra 1.0.x. I've spoken with other operators and they anecdotally
report the same behavior when doing the same operation you describe.
https://issues.apache.org/jira/browse/CASSANDRA-4446
The more people who report that they have the issue, the greater
chance of a response or fix, so I suggest commenting "me too!" on that
ticket.. :)
=Rob
--
=Robert Coli
AIM>ALK - rcoli@palominodb.com
YAHOO - rcoli.palominob
SKYPE - rcoli_palominodb