You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Radim Kolar <hs...@sendmail.cz> on 2011/11/08 07:58:12 UTC
Re: shutdown by KILL
> For things like rolling restarts, we do:
>
> disablethrift
> disablegossip
> (...wait for all nodes to see this node go down..)
> drain
i implemented this in our batch scripts for cassandra
disablegossip
sleep 10 seconds
dissablethrift
drain
KILL -TERM
similar thing should be added to bin/stop-server
Re: shutdown by KILL
Posted by Radim Kolar <hs...@sendmail.cz>.
> that's why disabling gossip + flush is better than drain. we should
> probably remove it.
drain could be good if there is way to undrain node - to switch it back
into r/w.
Implement nodetool shutdown which will work like we are trying. First
stop gossip then wait for other nodes to see it down, then disable
thrift, wait for local transactions to finish, flush + exit jvm
Re: shutdown by KILL
Posted by Jonathan Ellis <jb...@gmail.com>.
that's why disabling gossip + flush is better than drain. we should
probably remove it.
On Tue, Nov 8, 2011 at 6:38 PM, Radim Kolar <hs...@sendmail.cz> wrote:
> drain switches node into read only, which is undesirable because other nodes
> still see it up.
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: shutdown by KILL
Posted by Radim Kolar <hs...@sendmail.cz>.
drain switches node into read only, which is undesirable because other
nodes still see it up.
Re: shutdown by KILL
Posted by Jonathan Ellis <jb...@gmail.com>.
Kind of. That was the original design, and it's definitely still safe
to use that way, but the restart will obvoiusly be faster if it
doesn't have to replay commitlog. So a lot of people deploy something
like what Chris described.
On Tue, Nov 8, 2011 at 6:31 PM, Dan Hendry <da...@gmail.com> wrote:
> I *thought* Cassandra was supposed to have a crash only design[1]. My understanding is that it is safe to simply kill the process and with the regular TERM signal and, shutdown would be blocked on fsyncing the commit log but nothing else (obviously not true if you kill -9 the sucker) even when using periodic commit log mode. Again, my understanding is that the disable thrift/gossip commands have no functional purpose (both are going to go down with as part of drain are they not?). Issuing a 'drain' only minimizes the amount of commit log which needs to be replayed when starting back up.
>
> [1] Best reference I could find for 'crash only': http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Stopping-Cassandra-gracefully-td3865563.html
>
> Dan
>
> -----Original Message-----
> From: Radim Kolar [mailto:hsn@sendmail.cz]
> Sent: November-08-11 6:29
> To: user@cassandra.apache.org
> Subject: Re: shutdown by KILL
>
>> Ooops, sorry about that. I overlooked the drain. Sorry for the misinformation!
>>
> cassandra still replays log file even on clean shutdown via nodetool
> drain. It usually takes a while. I don't think it has concept of
> clean-shutdown like SQL databases.
> No virus found in this incoming message.
> Checked by AVG - www.avg.com
> Version: 9.0.920 / Virus Database: 271.1.1/4004 - Release Date: 11/08/11 02:34:00
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
RE: shutdown by KILL
Posted by Dan Hendry <da...@gmail.com>.
I *thought* Cassandra was supposed to have a crash only design[1]. My understanding is that it is safe to simply kill the process and with the regular TERM signal and, shutdown would be blocked on fsyncing the commit log but nothing else (obviously not true if you kill -9 the sucker) even when using periodic commit log mode. Again, my understanding is that the disable thrift/gossip commands have no functional purpose (both are going to go down with as part of drain are they not?). Issuing a 'drain' only minimizes the amount of commit log which needs to be replayed when starting back up.
[1] Best reference I could find for 'crash only': http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Stopping-Cassandra-gracefully-td3865563.html
Dan
-----Original Message-----
From: Radim Kolar [mailto:hsn@sendmail.cz]
Sent: November-08-11 6:29
To: user@cassandra.apache.org
Subject: Re: shutdown by KILL
> Ooops, sorry about that. I overlooked the drain. Sorry for the misinformation!
>
cassandra still replays log file even on clean shutdown via nodetool
drain. It usually takes a while. I don't think it has concept of
clean-shutdown like SQL databases.
No virus found in this incoming message.
Checked by AVG - www.avg.com
Version: 9.0.920 / Virus Database: 271.1.1/4004 - Release Date: 11/08/11 02:34:00
Re: shutdown by KILL
Posted by Radim Kolar <hs...@sendmail.cz>.
> Ooops, sorry about that. I overlooked the drain. Sorry for the misinformation!
>
cassandra still replays log file even on clean shutdown via nodetool
drain. It usually takes a while. I don't think it has concept of
clean-shutdown like SQL databases.
Re: shutdown by KILL
Posted by Peter Schuller <pe...@infidyne.com>.
> nodetool drain flushes memtables before returning. its probably safe to kill
> it after drain without waiting for commit log flush.
Ooops, sorry about that. I overlooked the drain. Sorry for the misinformation!
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)
Re: shutdown by KILL
Posted by Radim Kolar <hs...@sendmail.cz>.
> THEN, after that point, you have to wait for the 10 seconds to be
> reasonably sure the commit log has been flushed.
nodetool drain flushes memtables before returning. its probably safe to
kill it after drain without waiting for commit log flush.
Re: shutdown by KILL
Posted by Peter Schuller <pe...@infidyne.com>.
> sleep 10 seconds
If you are trying to prevent writes being lost in the typical case,
you need to wait *first* for all nodes to understand that the node is
down (as Chris mentioned). At that point, no node should be sending it
new writes. Assuming you also disabled the thrift interface, no writes
will be submitted locally either.
THEN, after that point, you have to wait for the 10 seconds to be
reasonably sure the commit log has been flushed. I'd wait more, say 12
seconds, just to have some margin in case the node is overloaded.
Still, just be aware there's no *guarantee*. E.g., if the commit log
writer is stalled due to disk saturation, data older than 15 seconds
may still be un-flushed at the point of kill. The way to get a
"guarantee" is to use batchwise sync.
If you're doing QUORUM you probably don't care, but if you're doing
ONE and want an actual *guarantee* that writes aren't lost, you might.
But on the other hand if you want a guarantee you shouldn't be using
ONE anyway since a node can crash at any time.
--
/ Peter Schuller (@scode, http://worldmodscode.wordpress.com)