You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Javier Canillas <ja...@gmail.com> on 2017/10/12 17:24:26 UTC
Best approach to prepare to shutdown a cassandra node
Hello everyone,
I have some time working with Cassandra, but every time I need to shutdown
a node (for any reason like upgrading version or moving instance to another
host) I see several errors on the client applications (yes, I'm using the
official java driver).
By the way, I'm starting C* as a stand-alone process
<https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
and C* version is 3.11.0.
The way I have implemented the shutdown process is something like the
following:
*# Drain all information from commitlog into sstables*
*bin/nodetool drain*
*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
* echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
cassandra)"*
* kill $cassandra_pid*
* echo -n "+ Checking it is down. "*
* counter=10*
* while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
2>&1 ]*
* do*
* echo -n ". "*
* ((counter--))*
* sleep 1s*
* done*
* echo ""*
* if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
* echo "+ Its down."*
* else*
* echo "- Killing Cassandra."*
* kill -9 $cassandra_pid*
* fi*
*else*
* echo "Care there was a problem finding Cassandra PID"*
*fi*
Should I add at the beginning the following lines?
echo "shutdowing cassandra gracefully with: nodetool disable gossip"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
echo "shutdowing cassandra gracefully with: nodetool disable binary
protocol"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
echo "shutdowing cassandra gracefully with: nodetool thrift"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
The shutdown log is the following:
*WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
StorageService.java:321 - Stopping gossip by operator request*
*INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
Gossiper.java:1532 - Announcing shutdown*
*INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
jump to shutdown*
*INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
Server.java:176 - Stop listening for CQL clients*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
StorageService.java:1442 - DRAINING: starting drain process*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
HintsService.java:220 - Paused hints dispatch*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
Gossiper.java:1532 - Announcing shutdown*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
shutdown*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
MessagingService.java:984 - Waiting for messaging service to quiesce*
*INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
the accept() thread*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
HintsService.java:220 - Paused hints dispatch*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
StorageService.java:1442 - DRAINED*
Disabling Gossip seemed a good idea, but watching the logs, it may use it
to gracefully telling the other nodes he is going down, so I don't know if
it's good or bad idea.
Disabling Thrift and Binary protocol should only avoid new connections, but
the one stablished and running should be attempted to finish.
Any thoughts or comments?
Thanks
Javier.
Re: Best approach to prepare to shutdown a cassandra node
Posted by Anshu Vajpayee <an...@gmail.com>.
use nodetool stopdaemon
On Wed, Oct 25, 2017 at 4:42 AM, Javier Canillas <ja...@gmail.com>
wrote:
> So, just to clarify.. a good approach to shutdown an instance of cassandra
> should be:
>
> *# Drain all information from commitlog into sstables*
> *bin/nodetool stopdaemon*
> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
> '{print $2}'`*
> *if [ "$?" -ne 0 ]; then*
> * echo "Cassandra stopdaemon fail? Please check logs"*
> * if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ];
> then*
> * echo "Cassandra is still running, killing it gracefully"*
> * kill $cassandra_pid*
> * echo -n "+ Checking it is down. "*
> * counter=10*
> * while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
> /dev/null 2>&1 ]*
> * do*
> * echo -n ". "*
> * ((counter--))*
> * sleep 1s*
> * done*
> * echo ""*
> * if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
> * echo "+ Its down."*
> * else*
> * echo "- Killing forcefully Cassandra."*
> * kill -9 $cassandra_pid*
> * fi*
> * else*
> * echo "Care there was a problem finding Cassandra PID, it
> might be still running"*
> * exit 1*
> * fi*
> * else*
> * echo "Cassandra stopped"*
> *fi*
>
> 2017-10-20 9:04 GMT-03:00 Lutaya Shafiq Holmes <lu...@gmail.com>:
>
>> Looking at the code in trunk, the stopdemon command invokes the
>> CassandraDaemon.stop() function which does a graceful shutdown by
>> stopping jmxServer and drains the node by the shutdown hook.
>>
>>
>> On 10/20/17, Simon Fontana Oscarsson
>> <si...@ericsson.com> wrote:
>> > Yes, drain will always be run when Cassandra exits normally.
>> >
>> > On 2017-10-20 00:57, Varun Gupta wrote:
>> >> Does, nodetool stopdaemon, implicitly drain too? or we should invoke
>> >> drain and then stopdaemon?
>> >>
>> >> On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson
>> >> <simon.fontana.oscarsson@ericsson.com
>> >> <ma...@ericsson.com>> wrote:
>> >>
>> >> Looking at the code in trunk, the stopdemon command invokes the
>> >> CassandraDaemon.stop() function which does a graceful shutdown by
>> >> stopping jmxServer and drains the node by the shutdown hook.
>> >>
>> >> /Simon
>> >>
>> >>
>> >> On 2017-10-13 20:42, Javier Canillas wrote:
>> >>> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>> >>>
>> >>> Or did it change?
>> >>>
>> >>> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee
>> >>> <anshu.vajpayee@gmail.com <ma...@gmail.com>>:
>> >>>
>> >>> Why are you killing when we have nodetool stopdaemon ?
>> >>>
>> >>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
>> >>> <javier.canillas@gmail.com
>> >>> <ma...@gmail.com>> wrote:
>> >>>
>> >>> That's what I thought.
>> >>>
>> >>> Thanks!
>> >>>
>> >>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger
>> >>> <hkroger@gmail.com <ma...@gmail.com>>:
>> >>>
>> >>> Hi,
>> >>>
>> >>> Drain should be enough. It stops accepting writes
>> >>> and after that cassandra can be safely shut down.
>> >>>
>> >>> Hannu
>> >>>
>> >>> On 12 October 2017 at 20:24:41, Javier Canillas
>> >>> (javier.canillas@gmail.com
>> >>> <ma...@gmail.com>) wrote:
>> >>>
>> >>>> Hello everyone,
>> >>>>
>> >>>> I have some time working with Cassandra, but every
>> >>>> time I need to shutdown a node (for any reason like
>> >>>> upgrading version or moving instance to another
>> >>>> host) I see several errors on the client
>> >>>> applications (yes, I'm using the official java
>> driver).
>> >>>>
>> >>>> By the way, I'm starting C* as a stand-alone process
>> >>>>
>> >>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initia
>> lize/referenceStartCprocess.html?hl=start>,
>> >>>> and C* version is 3.11.0.
>> >>>>
>> >>>> The way I have implemented the shutdown process is
>> >>>> something like the following:
>> >>>>
>> >>>> /# Drain all information from commitlog into
>> sstables/
>> >>>> /bin/nodetool drain
>> >>>> /
>> >>>> /
>> >>>> /
>> >>>> /cassandra_pid=`ps -ef|grep
>> >>>> "java.*apache-cassandra"|grep -v "grep"|awk '{print
>> >>>> $2}'`
>> >>>> /
>> >>>> /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid"
>> >>>> -ne "1" ]; then/
>> >>>> / echo "Asking Cassandra to shutdown (nodetool
>> >>>> drain doesn't stop cassandra)"/
>> >>>> / kill $cassandra_pid/
>> >>>> /
>> >>>> /
>> >>>> / echo -n "+ Checking it is down. "/
>> >>>> / counter=10/
>> >>>> / while [ "$counter" -ne 0 -a ! kill -0
>> >>>> $cassandra_pid > /dev/null 2>&1 ]/
>> >>>> / do/
>> >>>> / echo -n ". "/
>> >>>> /((counter--))/
>> >>>> /sleep 1s/
>> >>>> / done/
>> >>>> / echo ""/
>> >>>> / if ! kill -0 $cassandra_pid > /dev/null 2>&1;
>> then/
>> >>>> / echo "+ Its down."/
>> >>>> / else/
>> >>>> / echo "- Killing Cassandra."/
>> >>>> / kill -9 $cassandra_pid/
>> >>>> / fi/
>> >>>> /else/
>> >>>> / echo "Care there was a problem finding Cassandra
>> >>>> PID"/
>> >>>> /fi/
>> >>>> /
>> >>>> /
>> >>>> Should I add at the beginning the following lines?
>> >>>>
>> >>>> echo "shutdowing cassandra gracefully with: nodetool
>> >>>> disable gossip"
>> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>> >>>> disablegossip
>> >>>> echo "shutdowing cassandra gracefully with: nodetool
>> >>>> disable binary protocol"
>> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>> >>>> disablebinary
>> >>>> echo "shutdowing cassandra gracefully with: nodetool
>> >>>> thrift"
>> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>> >>>> disablethrift
>> >>>>
>> >>>> The shutdown log is the following:
>> >>>>
>> >>>> /WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> >>>> 14:20:52,343 StorageService.java:321 - Stopping
>> >>>> gossip by operator request/
>> >>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> >>>> 14:20:52,344 Gossiper.java:1532 - Announcing
>> shutdown/
>> >>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> >>>> 14:20:52,355 StorageService.java:2268 - Node
>> >>>> /10.254.169.36 <http://10.254.169.36> state jump to
>> >>>> shutdown/
>> >>>> /INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
>> >>>> 14:20:56,141 Server.java:176 - Stop listening for
>> >>>> CQL clients/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:20:59,472 StorageService.java:1442 - DRAINING:
>> >>>> starting drain process/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:20:59,474 HintsService.java:220 - Paused hints
>> >>>> dispatch/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:20:59,477 Gossiper.java:1532 - Announcing
>> shutdown/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:20:59,480 StorageService.java:2268 - Node
>> >>>> /127.0.0.1 <http://127.0.0.1> state jump to
>> shutdown/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:21:01,483 MessagingService.java:984 - Waiting for
>> >>>> messaging service to quiesce/
>> >>>> /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
>> >>>> 2017-10-12 14:21:01,485 MessagingService.java:1338 -
>> >>>> MessagingService has terminated the accept() thread/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:21:02,095 HintsService.java:220 - Paused hints
>> >>>> dispatch/
>> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> >>>> 14:21:02,111 StorageService.java:1442 - DRAINED/
>> >>>>
>> >>>> Disabling Gossip seemed a good idea, but watching
>> >>>> the logs, it may use it to gracefully telling the
>> >>>> other nodes he is going down, so I don't know if
>> >>>> it's good or bad idea.
>> >>>>
>> >>>> Disabling Thrift and Binary protocol should only
>> >>>> avoid new connections, but the one stablished and
>> >>>> running should be attempted to finish.
>> >>>>
>> >>>> Any thoughts or comments?
>> >>>>
>> >>>> Thanks
>> >>>>
>> >>>> Javier.
>> >>>>
>> >>>>
>> >>>
>> >>>
>> >>>
>> >>>
>> >>> --
>> >>> *Regards,*
>> >>> *Anshu *
>> >>> *
>> >>> *
>> >>>
>> >>>
>> >>
>> >>
>> >
>> >
>>
>>
>> --
>> Lutaaya Shafiq
>> Web: www.ronzag.com | info@ronzag.com
>> Mobile: +256702772721 <+256%20702%20772721> | +256783564130
>> <+256%20783%20564130>
>> Twitter: @lutayashafiq
>> Skype: lutaya5
>> Blog: lutayashafiq.com
>> http://www.fourcornersalliancegroup.com/?a=shafiqholmes
>>
>> "The most beautiful people we have known are those who have known defeat,
>> known suffering, known struggle, known loss and have found their way out
>> of
>> the depths. These persons have an appreciation, a sensitivity and an
>> understanding of life that fills them with compassion, gentleness and a
>> deep loving concern. Beautiful people do not just happen." - *Elisabeth
>> Kubler-Ross*
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
>> For additional commands, e-mail: user-help@cassandra.apache.org
>>
>>
>
--
*Regards,*
*Anshu *
Re: Best approach to prepare to shutdown a cassandra node
Posted by Javier Canillas <ja...@gmail.com>.
So, just to clarify.. a good approach to shutdown an instance of cassandra
should be:
*# Drain all information from commitlog into sstables*
*bin/nodetool stopdaemon*
*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ "$?" -ne 0 ]; then*
* echo "Cassandra stopdaemon fail? Please check logs"*
* if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
* echo "Cassandra is still running, killing it gracefully"*
* kill $cassandra_pid*
* echo -n "+ Checking it is down. "*
* counter=10*
* while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
/dev/null 2>&1 ]*
* do*
* echo -n ". "*
* ((counter--))*
* sleep 1s*
* done*
* echo ""*
* if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
* echo "+ Its down."*
* else*
* echo "- Killing forcefully Cassandra."*
* kill -9 $cassandra_pid*
* fi*
* else*
* echo "Care there was a problem finding Cassandra PID, it
might be still running"*
* exit 1*
* fi*
* else*
* echo "Cassandra stopped"*
*fi*
2017-10-20 9:04 GMT-03:00 Lutaya Shafiq Holmes <lu...@gmail.com>:
> Looking at the code in trunk, the stopdemon command invokes the
> CassandraDaemon.stop() function which does a graceful shutdown by
> stopping jmxServer and drains the node by the shutdown hook.
>
>
> On 10/20/17, Simon Fontana Oscarsson
> <si...@ericsson.com> wrote:
> > Yes, drain will always be run when Cassandra exits normally.
> >
> > On 2017-10-20 00:57, Varun Gupta wrote:
> >> Does, nodetool stopdaemon, implicitly drain too? or we should invoke
> >> drain and then stopdaemon?
> >>
> >> On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson
> >> <simon.fontana.oscarsson@ericsson.com
> >> <ma...@ericsson.com>> wrote:
> >>
> >> Looking at the code in trunk, the stopdemon command invokes the
> >> CassandraDaemon.stop() function which does a graceful shutdown by
> >> stopping jmxServer and drains the node by the shutdown hook.
> >>
> >> /Simon
> >>
> >>
> >> On 2017-10-13 20:42, Javier Canillas wrote:
> >>> As far as I know, the nodetool stopdaemon is doing a "kill -9".
> >>>
> >>> Or did it change?
> >>>
> >>> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee
> >>> <anshu.vajpayee@gmail.com <ma...@gmail.com>>:
> >>>
> >>> Why are you killing when we have nodetool stopdaemon ?
> >>>
> >>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
> >>> <javier.canillas@gmail.com
> >>> <ma...@gmail.com>> wrote:
> >>>
> >>> That's what I thought.
> >>>
> >>> Thanks!
> >>>
> >>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger
> >>> <hkroger@gmail.com <ma...@gmail.com>>:
> >>>
> >>> Hi,
> >>>
> >>> Drain should be enough. It stops accepting writes
> >>> and after that cassandra can be safely shut down.
> >>>
> >>> Hannu
> >>>
> >>> On 12 October 2017 at 20:24:41, Javier Canillas
> >>> (javier.canillas@gmail.com
> >>> <ma...@gmail.com>) wrote:
> >>>
> >>>> Hello everyone,
> >>>>
> >>>> I have some time working with Cassandra, but every
> >>>> time I need to shutdown a node (for any reason like
> >>>> upgrading version or moving instance to another
> >>>> host) I see several errors on the client
> >>>> applications (yes, I'm using the official java
> driver).
> >>>>
> >>>> By the way, I'm starting C* as a stand-alone process
> >>>>
> >>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/
> referenceStartCprocess.html?hl=start>,
> >>>> and C* version is 3.11.0.
> >>>>
> >>>> The way I have implemented the shutdown process is
> >>>> something like the following:
> >>>>
> >>>> /# Drain all information from commitlog into sstables/
> >>>> /bin/nodetool drain
> >>>> /
> >>>> /
> >>>> /
> >>>> /cassandra_pid=`ps -ef|grep
> >>>> "java.*apache-cassandra"|grep -v "grep"|awk '{print
> >>>> $2}'`
> >>>> /
> >>>> /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid"
> >>>> -ne "1" ]; then/
> >>>> / echo "Asking Cassandra to shutdown (nodetool
> >>>> drain doesn't stop cassandra)"/
> >>>> / kill $cassandra_pid/
> >>>> /
> >>>> /
> >>>> / echo -n "+ Checking it is down. "/
> >>>> / counter=10/
> >>>> / while [ "$counter" -ne 0 -a ! kill -0
> >>>> $cassandra_pid > /dev/null 2>&1 ]/
> >>>> / do/
> >>>> / echo -n ". "/
> >>>> /((counter--))/
> >>>> /sleep 1s/
> >>>> / done/
> >>>> / echo ""/
> >>>> / if ! kill -0 $cassandra_pid > /dev/null 2>&1; then/
> >>>> / echo "+ Its down."/
> >>>> / else/
> >>>> / echo "- Killing Cassandra."/
> >>>> / kill -9 $cassandra_pid/
> >>>> / fi/
> >>>> /else/
> >>>> / echo "Care there was a problem finding Cassandra
> >>>> PID"/
> >>>> /fi/
> >>>> /
> >>>> /
> >>>> Should I add at the beginning the following lines?
> >>>>
> >>>> echo "shutdowing cassandra gracefully with: nodetool
> >>>> disable gossip"
> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
> >>>> disablegossip
> >>>> echo "shutdowing cassandra gracefully with: nodetool
> >>>> disable binary protocol"
> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
> >>>> disablebinary
> >>>> echo "shutdowing cassandra gracefully with: nodetool
> >>>> thrift"
> >>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
> >>>> disablethrift
> >>>>
> >>>> The shutdown log is the following:
> >>>>
> >>>> /WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
> >>>> 14:20:52,343 StorageService.java:321 - Stopping
> >>>> gossip by operator request/
> >>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
> >>>> 14:20:52,344 Gossiper.java:1532 - Announcing shutdown/
> >>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
> >>>> 14:20:52,355 StorageService.java:2268 - Node
> >>>> /10.254.169.36 <http://10.254.169.36> state jump to
> >>>> shutdown/
> >>>> /INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
> >>>> 14:20:56,141 Server.java:176 - Stop listening for
> >>>> CQL clients/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:20:59,472 StorageService.java:1442 - DRAINING:
> >>>> starting drain process/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:20:59,474 HintsService.java:220 - Paused hints
> >>>> dispatch/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:20:59,477 Gossiper.java:1532 - Announcing shutdown/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:20:59,480 StorageService.java:2268 - Node
> >>>> /127.0.0.1 <http://127.0.0.1> state jump to shutdown/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:21:01,483 MessagingService.java:984 - Waiting for
> >>>> messaging service to quiesce/
> >>>> /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
> >>>> 2017-10-12 14:21:01,485 MessagingService.java:1338 -
> >>>> MessagingService has terminated the accept() thread/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:21:02,095 HintsService.java:220 - Paused hints
> >>>> dispatch/
> >>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
> >>>> 14:21:02,111 StorageService.java:1442 - DRAINED/
> >>>>
> >>>> Disabling Gossip seemed a good idea, but watching
> >>>> the logs, it may use it to gracefully telling the
> >>>> other nodes he is going down, so I don't know if
> >>>> it's good or bad idea.
> >>>>
> >>>> Disabling Thrift and Binary protocol should only
> >>>> avoid new connections, but the one stablished and
> >>>> running should be attempted to finish.
> >>>>
> >>>> Any thoughts or comments?
> >>>>
> >>>> Thanks
> >>>>
> >>>> Javier.
> >>>>
> >>>>
> >>>
> >>>
> >>>
> >>>
> >>> --
> >>> *Regards,*
> >>> *Anshu *
> >>> *
> >>> *
> >>>
> >>>
> >>
> >>
> >
> >
>
>
> --
> Lutaaya Shafiq
> Web: www.ronzag.com | info@ronzag.com
> Mobile: +256702772721 | +256783564130
> Twitter: @lutayashafiq
> Skype: lutaya5
> Blog: lutayashafiq.com
> http://www.fourcornersalliancegroup.com/?a=shafiqholmes
>
> "The most beautiful people we have known are those who have known defeat,
> known suffering, known struggle, known loss and have found their way out of
> the depths. These persons have an appreciation, a sensitivity and an
> understanding of life that fills them with compassion, gentleness and a
> deep loving concern. Beautiful people do not just happen." - *Elisabeth
> Kubler-Ross*
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Lutaya Shafiq Holmes <lu...@gmail.com>.
Looking at the code in trunk, the stopdemon command invokes the
CassandraDaemon.stop() function which does a graceful shutdown by
stopping jmxServer and drains the node by the shutdown hook.
On 10/20/17, Simon Fontana Oscarsson
<si...@ericsson.com> wrote:
> Yes, drain will always be run when Cassandra exits normally.
>
> On 2017-10-20 00:57, Varun Gupta wrote:
>> Does, nodetool stopdaemon, implicitly drain too? or we should invoke
>> drain and then stopdaemon?
>>
>> On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson
>> <simon.fontana.oscarsson@ericsson.com
>> <ma...@ericsson.com>> wrote:
>>
>> Looking at the code in trunk, the stopdemon command invokes the
>> CassandraDaemon.stop() function which does a graceful shutdown by
>> stopping jmxServer and drains the node by the shutdown hook.
>>
>> /Simon
>>
>>
>> On 2017-10-13 20:42, Javier Canillas wrote:
>>> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>>>
>>> Or did it change?
>>>
>>> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee
>>> <anshu.vajpayee@gmail.com <ma...@gmail.com>>:
>>>
>>> Why are you killing when we have nodetool stopdaemon ?
>>>
>>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
>>> <javier.canillas@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> That's what I thought.
>>>
>>> Thanks!
>>>
>>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger
>>> <hkroger@gmail.com <ma...@gmail.com>>:
>>>
>>> Hi,
>>>
>>> Drain should be enough. It stops accepting writes
>>> and after that cassandra can be safely shut down.
>>>
>>> Hannu
>>>
>>> On 12 October 2017 at 20:24:41, Javier Canillas
>>> (javier.canillas@gmail.com
>>> <ma...@gmail.com>) wrote:
>>>
>>>> Hello everyone,
>>>>
>>>> I have some time working with Cassandra, but every
>>>> time I need to shutdown a node (for any reason like
>>>> upgrading version or moving instance to another
>>>> host) I see several errors on the client
>>>> applications (yes, I'm using the official java driver).
>>>>
>>>> By the way, I'm starting C* as a stand-alone process
>>>>
>>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>>> and C* version is 3.11.0.
>>>>
>>>> The way I have implemented the shutdown process is
>>>> something like the following:
>>>>
>>>> /# Drain all information from commitlog into sstables/
>>>> /bin/nodetool drain
>>>> /
>>>> /
>>>> /
>>>> /cassandra_pid=`ps -ef|grep
>>>> "java.*apache-cassandra"|grep -v "grep"|awk '{print
>>>> $2}'`
>>>> /
>>>> /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid"
>>>> -ne "1" ]; then/
>>>> / echo "Asking Cassandra to shutdown (nodetool
>>>> drain doesn't stop cassandra)"/
>>>> / kill $cassandra_pid/
>>>> /
>>>> /
>>>> / echo -n "+ Checking it is down. "/
>>>> / counter=10/
>>>> / while [ "$counter" -ne 0 -a ! kill -0
>>>> $cassandra_pid > /dev/null 2>&1 ]/
>>>> / do/
>>>> / echo -n ". "/
>>>> /((counter--))/
>>>> /sleep 1s/
>>>> / done/
>>>> / echo ""/
>>>> / if ! kill -0 $cassandra_pid > /dev/null 2>&1; then/
>>>> / echo "+ Its down."/
>>>> / else/
>>>> / echo "- Killing Cassandra."/
>>>> / kill -9 $cassandra_pid/
>>>> / fi/
>>>> /else/
>>>> / echo "Care there was a problem finding Cassandra
>>>> PID"/
>>>> /fi/
>>>> /
>>>> /
>>>> Should I add at the beginning the following lines?
>>>>
>>>> echo "shutdowing cassandra gracefully with: nodetool
>>>> disable gossip"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>>> disablegossip
>>>> echo "shutdowing cassandra gracefully with: nodetool
>>>> disable binary protocol"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>>> disablebinary
>>>> echo "shutdowing cassandra gracefully with: nodetool
>>>> thrift"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>>> disablethrift
>>>>
>>>> The shutdown log is the following:
>>>>
>>>> /WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>>> 14:20:52,343 StorageService.java:321 - Stopping
>>>> gossip by operator request/
>>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>>> 14:20:52,344 Gossiper.java:1532 - Announcing shutdown/
>>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>>> 14:20:52,355 StorageService.java:2268 - Node
>>>> /10.254.169.36 <http://10.254.169.36> state jump to
>>>> shutdown/
>>>> /INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
>>>> 14:20:56,141 Server.java:176 - Stop listening for
>>>> CQL clients/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:20:59,472 StorageService.java:1442 - DRAINING:
>>>> starting drain process/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:20:59,474 HintsService.java:220 - Paused hints
>>>> dispatch/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:20:59,477 Gossiper.java:1532 - Announcing shutdown/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:20:59,480 StorageService.java:2268 - Node
>>>> /127.0.0.1 <http://127.0.0.1> state jump to shutdown/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:21:01,483 MessagingService.java:984 - Waiting for
>>>> messaging service to quiesce/
>>>> /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
>>>> 2017-10-12 14:21:01,485 MessagingService.java:1338 -
>>>> MessagingService has terminated the accept() thread/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:21:02,095 HintsService.java:220 - Paused hints
>>>> dispatch/
>>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>>> 14:21:02,111 StorageService.java:1442 - DRAINED/
>>>>
>>>> Disabling Gossip seemed a good idea, but watching
>>>> the logs, it may use it to gracefully telling the
>>>> other nodes he is going down, so I don't know if
>>>> it's good or bad idea.
>>>>
>>>> Disabling Thrift and Binary protocol should only
>>>> avoid new connections, but the one stablished and
>>>> running should be attempted to finish.
>>>>
>>>> Any thoughts or comments?
>>>>
>>>> Thanks
>>>>
>>>> Javier.
>>>>
>>>>
>>>
>>>
>>>
>>>
>>> --
>>> *Regards,*
>>> *Anshu *
>>> *
>>> *
>>>
>>>
>>
>>
>
>
--
Lutaaya Shafiq
Web: www.ronzag.com | info@ronzag.com
Mobile: +256702772721 | +256783564130
Twitter: @lutayashafiq
Skype: lutaya5
Blog: lutayashafiq.com
http://www.fourcornersalliancegroup.com/?a=shafiqholmes
"The most beautiful people we have known are those who have known defeat,
known suffering, known struggle, known loss and have found their way out of
the depths. These persons have an appreciation, a sensitivity and an
understanding of life that fills them with compassion, gentleness and a
deep loving concern. Beautiful people do not just happen." - *Elisabeth
Kubler-Ross*
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: Best approach to prepare to shutdown a cassandra node
Posted by Simon Fontana Oscarsson <si...@ericsson.com>.
Yes, drain will always be run when Cassandra exits normally.
On 2017-10-20 00:57, Varun Gupta wrote:
> Does, nodetool stopdaemon, implicitly drain too? or we should invoke
> drain and then stopdaemon?
>
> On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson
> <simon.fontana.oscarsson@ericsson.com
> <ma...@ericsson.com>> wrote:
>
> Looking at the code in trunk, the stopdemon command invokes the
> CassandraDaemon.stop() function which does a graceful shutdown by
> stopping jmxServer and drains the node by the shutdown hook.
>
> /Simon
>
>
> On 2017-10-13 20:42, Javier Canillas wrote:
>> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>>
>> Or did it change?
>>
>> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee
>> <anshu.vajpayee@gmail.com <ma...@gmail.com>>:
>>
>> Why are you killing when we have nodetool stopdaemon ?
>>
>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
>> <javier.canillas@gmail.com
>> <ma...@gmail.com>> wrote:
>>
>> That's what I thought.
>>
>> Thanks!
>>
>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger
>> <hkroger@gmail.com <ma...@gmail.com>>:
>>
>> Hi,
>>
>> Drain should be enough. It stops accepting writes
>> and after that cassandra can be safely shut down.
>>
>> Hannu
>>
>> On 12 October 2017 at 20:24:41, Javier Canillas
>> (javier.canillas@gmail.com
>> <ma...@gmail.com>) wrote:
>>
>>> Hello everyone,
>>>
>>> I have some time working with Cassandra, but every
>>> time I need to shutdown a node (for any reason like
>>> upgrading version or moving instance to another
>>> host) I see several errors on the client
>>> applications (yes, I'm using the official java driver).
>>>
>>> By the way, I'm starting C* as a stand-alone process
>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>> and C* version is 3.11.0.
>>>
>>> The way I have implemented the shutdown process is
>>> something like the following:
>>>
>>> /# Drain all information from commitlog into sstables/
>>> /bin/nodetool drain
>>> /
>>> /
>>> /
>>> /cassandra_pid=`ps -ef|grep
>>> "java.*apache-cassandra"|grep -v "grep"|awk '{print
>>> $2}'`
>>> /
>>> /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid"
>>> -ne "1" ]; then/
>>> / echo "Asking Cassandra to shutdown (nodetool
>>> drain doesn't stop cassandra)"/
>>> / kill $cassandra_pid/
>>> /
>>> /
>>> / echo -n "+ Checking it is down. "/
>>> / counter=10/
>>> / while [ "$counter" -ne 0 -a ! kill -0
>>> $cassandra_pid > /dev/null 2>&1 ]/
>>> / do/
>>> / echo -n ". "/
>>> /((counter--))/
>>> /sleep 1s/
>>> / done/
>>> / echo ""/
>>> / if ! kill -0 $cassandra_pid > /dev/null 2>&1; then/
>>> / echo "+ Its down."/
>>> / else/
>>> / echo "- Killing Cassandra."/
>>> / kill -9 $cassandra_pid/
>>> / fi/
>>> /else/
>>> / echo "Care there was a problem finding Cassandra
>>> PID"/
>>> /fi/
>>> /
>>> /
>>> Should I add at the beginning the following lines?
>>>
>>> echo "shutdowing cassandra gracefully with: nodetool
>>> disable gossip"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>> disablegossip
>>> echo "shutdowing cassandra gracefully with: nodetool
>>> disable binary protocol"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>> disablebinary
>>> echo "shutdowing cassandra gracefully with: nodetool
>>> thrift"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool
>>> disablethrift
>>>
>>> The shutdown log is the following:
>>>
>>> /WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>> 14:20:52,343 StorageService.java:321 - Stopping
>>> gossip by operator request/
>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>> 14:20:52,344 Gossiper.java:1532 - Announcing shutdown/
>>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>>> 14:20:52,355 StorageService.java:2268 - Node
>>> /10.254.169.36 <http://10.254.169.36> state jump to
>>> shutdown/
>>> /INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
>>> 14:20:56,141 Server.java:176 - Stop listening for
>>> CQL clients/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:20:59,472 StorageService.java:1442 - DRAINING:
>>> starting drain process/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:20:59,474 HintsService.java:220 - Paused hints
>>> dispatch/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:20:59,477 Gossiper.java:1532 - Announcing shutdown/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:20:59,480 StorageService.java:2268 - Node
>>> /127.0.0.1 <http://127.0.0.1> state jump to shutdown/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:21:01,483 MessagingService.java:984 - Waiting for
>>> messaging service to quiesce/
>>> /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
>>> 2017-10-12 14:21:01,485 MessagingService.java:1338 -
>>> MessagingService has terminated the accept() thread/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:21:02,095 HintsService.java:220 - Paused hints
>>> dispatch/
>>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>>> 14:21:02,111 StorageService.java:1442 - DRAINED/
>>>
>>> Disabling Gossip seemed a good idea, but watching
>>> the logs, it may use it to gracefully telling the
>>> other nodes he is going down, so I don't know if
>>> it's good or bad idea.
>>>
>>> Disabling Thrift and Binary protocol should only
>>> avoid new connections, but the one stablished and
>>> running should be attempted to finish.
>>>
>>> Any thoughts or comments?
>>>
>>> Thanks
>>>
>>> Javier.
>>>
>>>
>>
>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>> *
>> *
>>
>>
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Varun Gupta <va...@uber.com>.
Does, nodetool stopdaemon, implicitly drain too? or we should invoke drain
and then stopdaemon?
On Mon, Oct 16, 2017 at 4:54 AM, Simon Fontana Oscarsson <
simon.fontana.oscarsson@ericsson.com> wrote:
> Looking at the code in trunk, the stopdemon command invokes the
> CassandraDaemon.stop() function which does a graceful shutdown by stopping
> jmxServer and drains the node by the shutdown hook.
>
> /Simon
>
>
> On 2017-10-13 20:42, Javier Canillas wrote:
>
> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>
> Or did it change?
>
> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee <an...@gmail.com>:
>
>> Why are you killing when we have nodetool stopdaemon ?
>>
>> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas <
>> javier.canillas@gmail.com> wrote:
>>
>>> That's what I thought.
>>>
>>> Thanks!
>>>
>>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger <hk...@gmail.com>:
>>>
>>>> Hi,
>>>>
>>>> Drain should be enough. It stops accepting writes and after that
>>>> cassandra can be safely shut down.
>>>>
>>>> Hannu
>>>>
>>>> On 12 October 2017 at 20:24:41, Javier Canillas (
>>>> javier.canillas@gmail.com) wrote:
>>>>
>>>> Hello everyone,
>>>>
>>>> I have some time working with Cassandra, but every time I need to
>>>> shutdown a node (for any reason like upgrading version or moving instance
>>>> to another host) I see several errors on the client applications (yes, I'm
>>>> using the official java driver).
>>>>
>>>> By the way, I'm starting C* as a stand-alone process
>>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>>> and C* version is 3.11.0.
>>>>
>>>> The way I have implemented the shutdown process is something like the
>>>> following:
>>>>
>>>> *# Drain all information from commitlog into sstables*
>>>>
>>>> *bin/nodetool drain *
>>>>
>>>>
>>>> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
>>>> '{print $2}'` *
>>>> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
>>>> * echo "Asking Cassandra to shutdown (nodetool drain doesn't
>>>> stop cassandra)"*
>>>> * kill $cassandra_pid*
>>>>
>>>> * echo -n "+ Checking it is down. "*
>>>> * counter=10*
>>>> * while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
>>>> /dev/null 2>&1 ]*
>>>> * do*
>>>> * echo -n ". "*
>>>> * ((counter--))*
>>>> * sleep 1s*
>>>> * done*
>>>> * echo ""*
>>>> * if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
>>>> * echo "+ Its down."*
>>>> * else*
>>>> * echo "- Killing Cassandra."*
>>>> * kill -9 $cassandra_pid*
>>>> * fi*
>>>> *else*
>>>> * echo "Care there was a problem finding Cassandra PID"*
>>>> *fi*
>>>>
>>>> Should I add at the beginning the following lines?
>>>>
>>>> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>>>> echo "shutdowing cassandra gracefully with: nodetool disable binary
>>>> protocol"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>>>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>>>
>>>> The shutdown log is the following:
>>>>
>>>> *WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
>>>> StorageService.java:321 - Stopping gossip by operator request*
>>>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
>>>> Gossiper.java:1532 - Announcing shutdown*
>>>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
>>>> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
>>>> jump to shutdown*
>>>> *INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
>>>> Server.java:176 - Stop listening for CQL clients*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
>>>> StorageService.java:1442 - DRAINING: starting drain process*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
>>>> HintsService.java:220 - Paused hints dispatch*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
>>>> Gossiper.java:1532 - Announcing shutdown*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
>>>> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
>>>> shutdown*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
>>>> MessagingService.java:984 - Waiting for messaging service to quiesce*
>>>> *INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
>>>> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
>>>> the accept() thread*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
>>>> HintsService.java:220 - Paused hints dispatch*
>>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
>>>> StorageService.java:1442 - DRAINED*
>>>>
>>>> Disabling Gossip seemed a good idea, but watching the logs, it may use
>>>> it to gracefully telling the other nodes he is going down, so I don't know
>>>> if it's good or bad idea.
>>>>
>>>> Disabling Thrift and Binary protocol should only avoid new connections,
>>>> but the one stablished and running should be attempted to finish.
>>>>
>>>> Any thoughts or comments?
>>>>
>>>> Thanks
>>>>
>>>> Javier.
>>>>
>>>>
>>>>
>>>
>>
>>
>> --
>> *Regards,*
>> *Anshu *
>>
>>
>>
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Simon Fontana Oscarsson <si...@ericsson.com>.
Looking at the code in trunk, the stopdemon command invokes the
CassandraDaemon.stop() function which does a graceful shutdown by
stopping jmxServer and drains the node by the shutdown hook.
/Simon
On 2017-10-13 20:42, Javier Canillas wrote:
> As far as I know, the nodetool stopdaemon is doing a "kill -9".
>
> Or did it change?
>
> 2017-10-12 23:49 GMT-03:00 Anshu Vajpayee <anshu.vajpayee@gmail.com
> <ma...@gmail.com>>:
>
> Why are you killing when we have nodetool stopdaemon ?
>
> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas
> <javier.canillas@gmail.com <ma...@gmail.com>> wrote:
>
> That's what I thought.
>
> Thanks!
>
> 2017-10-12 14:26 GMT-03:00 Hannu Kröger <hkroger@gmail.com
> <ma...@gmail.com>>:
>
> Hi,
>
> Drain should be enough. It stops accepting writes and
> after that cassandra can be safely shut down.
>
> Hannu
>
> On 12 October 2017 at 20:24:41, Javier Canillas
> (javier.canillas@gmail.com
> <ma...@gmail.com>) wrote:
>
>> Hello everyone,
>>
>> I have some time working with Cassandra, but every time I
>> need to shutdown a node (for any reason like upgrading
>> version or moving instance to another host) I see several
>> errors on the client applications (yes, I'm using the
>> official java driver).
>>
>> By the way, I'm starting C* as a stand-alone process
>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>> and C* version is 3.11.0.
>>
>> The way I have implemented the shutdown process is
>> something like the following:
>>
>> /# Drain all information from commitlog into sstables/
>> /bin/nodetool drain
>> /
>> /
>> /
>> /cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep
>> -v "grep"|awk '{print $2}'`
>> /
>> /if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne
>> "1" ]; then/
>> / echo "Asking Cassandra to shutdown (nodetool
>> drain doesn't stop cassandra)"/
>> / kill $cassandra_pid/
>> /
>> /
>> / echo -n "+ Checking it is down. "/
>> /counter=10/
>> / while [ "$counter" -ne 0 -a ! kill -0
>> $cassandra_pid > /dev/null 2>&1 ]/
>> / do/
>> /echo -n ". "/
>> /((counter--))/
>> /sleep 1s/
>> / done/
>> / echo ""/
>> / if ! kill -0 $cassandra_pid > /dev/null 2>&1; then/
>> /echo "+ Its down."/
>> / else/
>> /echo "- Killing Cassandra."/
>> /kill -9 $cassandra_pid/
>> / fi/
>> /else/
>> / echo "Care there was a problem finding Cassandra
>> PID"/
>> /fi/
>> /
>> /
>> Should I add at the beginning the following lines?
>>
>> echo "shutdowing cassandra gracefully with: nodetool
>> disable gossip"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>> echo "shutdowing cassandra gracefully with: nodetool
>> disable binary protocol"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>
>> The shutdown log is the following:
>>
>> /WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> 14:20:52,343 StorageService.java:321 - Stopping gossip by
>> operator request/
>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> 14:20:52,344 Gossiper.java:1532 - Announcing shutdown/
>> /INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12
>> 14:20:52,355 StorageService.java:2268 - Node
>> /10.254.169.36 <http://10.254.169.36> state jump to shutdown/
>> /INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12
>> 14:20:56,141 Server.java:176 - Stop listening for CQL
>> clients/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:20:59,472 StorageService.java:1442 - DRAINING:
>> starting drain process/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:20:59,474 HintsService.java:220 - Paused hints dispatch/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:20:59,477 Gossiper.java:1532 - Announcing shutdown/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:20:59,480 StorageService.java:2268 - Node /127.0.0.1
>> <http://127.0.0.1> state jump to shutdown/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:21:01,483 MessagingService.java:984 - Waiting for
>> messaging service to quiesce/
>> /INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>]
>> 2017-10-12 14:21:01,485 MessagingService.java:1338 -
>> MessagingService has terminated the accept() thread/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:21:02,095 HintsService.java:220 - Paused hints dispatch/
>> /INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12
>> 14:21:02,111 StorageService.java:1442 - DRAINED/
>>
>> Disabling Gossip seemed a good idea, but watching the
>> logs, it may use it to gracefully telling the other nodes
>> he is going down, so I don't know if it's good or bad idea.
>>
>> Disabling Thrift and Binary protocol should only avoid
>> new connections, but the one stablished and running
>> should be attempted to finish.
>>
>> Any thoughts or comments?
>>
>> Thanks
>>
>> Javier.
>>
>>
>
>
>
>
> --
> *Regards,*
> *Anshu *
> *
> *
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Javier Canillas <ja...@gmail.com>.
As far as I know, the nodetool stopdaemon is doing a "kill -9".
Or did it change?
2017-10-12 23:49 GMT-03:00 Anshu Vajpayee <an...@gmail.com>:
> Why are you killing when we have nodetool stopdaemon ?
>
> On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas <
> javier.canillas@gmail.com> wrote:
>
>> That's what I thought.
>>
>> Thanks!
>>
>> 2017-10-12 14:26 GMT-03:00 Hannu Kröger <hk...@gmail.com>:
>>
>>> Hi,
>>>
>>> Drain should be enough. It stops accepting writes and after that
>>> cassandra can be safely shut down.
>>>
>>> Hannu
>>>
>>> On 12 October 2017 at 20:24:41, Javier Canillas (
>>> javier.canillas@gmail.com) wrote:
>>>
>>> Hello everyone,
>>>
>>> I have some time working with Cassandra, but every time I need to
>>> shutdown a node (for any reason like upgrading version or moving instance
>>> to another host) I see several errors on the client applications (yes, I'm
>>> using the official java driver).
>>>
>>> By the way, I'm starting C* as a stand-alone process
>>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>>> and C* version is 3.11.0.
>>>
>>> The way I have implemented the shutdown process is something like the
>>> following:
>>>
>>> *# Drain all information from commitlog into sstables*
>>>
>>> *bin/nodetool drain*
>>>
>>>
>>> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
>>> '{print $2}'`*
>>> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
>>> * echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
>>> cassandra)"*
>>> * kill $cassandra_pid*
>>>
>>> * echo -n "+ Checking it is down. "*
>>> * counter=10*
>>> * while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid >
>>> /dev/null 2>&1 ]*
>>> * do*
>>> * echo -n ". "*
>>> * ((counter--))*
>>> * sleep 1s*
>>> * done*
>>> * echo ""*
>>> * if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
>>> * echo "+ Its down."*
>>> * else*
>>> * echo "- Killing Cassandra."*
>>> * kill -9 $cassandra_pid*
>>> * fi*
>>> *else*
>>> * echo "Care there was a problem finding Cassandra PID"*
>>> *fi*
>>>
>>> Should I add at the beginning the following lines?
>>>
>>> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>>> echo "shutdowing cassandra gracefully with: nodetool disable binary
>>> protocol"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>>
>>> The shutdown log is the following:
>>>
>>> *WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
>>> StorageService.java:321 - Stopping gossip by operator request*
>>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
>>> Gossiper.java:1532 - Announcing shutdown*
>>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
>>> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
>>> jump to shutdown*
>>> *INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
>>> Server.java:176 - Stop listening for CQL clients*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
>>> StorageService.java:1442 - DRAINING: starting drain process*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
>>> HintsService.java:220 - Paused hints dispatch*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
>>> Gossiper.java:1532 - Announcing shutdown*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
>>> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
>>> shutdown*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
>>> MessagingService.java:984 - Waiting for messaging service to quiesce*
>>> *INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
>>> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
>>> the accept() thread*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
>>> HintsService.java:220 - Paused hints dispatch*
>>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
>>> StorageService.java:1442 - DRAINED*
>>>
>>> Disabling Gossip seemed a good idea, but watching the logs, it may use
>>> it to gracefully telling the other nodes he is going down, so I don't know
>>> if it's good or bad idea.
>>>
>>> Disabling Thrift and Binary protocol should only avoid new connections,
>>> but the one stablished and running should be attempted to finish.
>>>
>>> Any thoughts or comments?
>>>
>>> Thanks
>>>
>>> Javier.
>>>
>>>
>>>
>>
>
>
> --
> *Regards,*
> *Anshu *
>
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Anshu Vajpayee <an...@gmail.com>.
Why are you killing when we have nodetool stopdaemon ?
On Fri, Oct 13, 2017 at 1:49 AM, Javier Canillas <ja...@gmail.com>
wrote:
> That's what I thought.
>
> Thanks!
>
> 2017-10-12 14:26 GMT-03:00 Hannu Kröger <hk...@gmail.com>:
>
>> Hi,
>>
>> Drain should be enough. It stops accepting writes and after that
>> cassandra can be safely shut down.
>>
>> Hannu
>>
>> On 12 October 2017 at 20:24:41, Javier Canillas (
>> javier.canillas@gmail.com) wrote:
>>
>> Hello everyone,
>>
>> I have some time working with Cassandra, but every time I need to
>> shutdown a node (for any reason like upgrading version or moving instance
>> to another host) I see several errors on the client applications (yes, I'm
>> using the official java driver).
>>
>> By the way, I'm starting C* as a stand-alone process
>> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
>> and C* version is 3.11.0.
>>
>> The way I have implemented the shutdown process is something like the
>> following:
>>
>> *# Drain all information from commitlog into sstables*
>>
>> *bin/nodetool drain*
>>
>>
>> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
>> '{print $2}'`*
>> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
>> * echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
>> cassandra)"*
>> * kill $cassandra_pid*
>>
>> * echo -n "+ Checking it is down. "*
>> * counter=10*
>> * while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
>> 2>&1 ]*
>> * do*
>> * echo -n ". "*
>> * ((counter--))*
>> * sleep 1s*
>> * done*
>> * echo ""*
>> * if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
>> * echo "+ Its down."*
>> * else*
>> * echo "- Killing Cassandra."*
>> * kill -9 $cassandra_pid*
>> * fi*
>> *else*
>> * echo "Care there was a problem finding Cassandra PID"*
>> *fi*
>>
>> Should I add at the beginning the following lines?
>>
>> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
>> echo "shutdowing cassandra gracefully with: nodetool disable binary
>> protocol"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
>> echo "shutdowing cassandra gracefully with: nodetool thrift"
>> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>>
>> The shutdown log is the following:
>>
>> *WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
>> StorageService.java:321 - Stopping gossip by operator request*
>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
>> Gossiper.java:1532 - Announcing shutdown*
>> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
>> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
>> jump to shutdown*
>> *INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
>> Server.java:176 - Stop listening for CQL clients*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
>> StorageService.java:1442 - DRAINING: starting drain process*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
>> HintsService.java:220 - Paused hints dispatch*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
>> Gossiper.java:1532 - Announcing shutdown*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
>> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
>> shutdown*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
>> MessagingService.java:984 - Waiting for messaging service to quiesce*
>> *INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
>> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
>> the accept() thread*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
>> HintsService.java:220 - Paused hints dispatch*
>> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
>> StorageService.java:1442 - DRAINED*
>>
>> Disabling Gossip seemed a good idea, but watching the logs, it may use it
>> to gracefully telling the other nodes he is going down, so I don't know if
>> it's good or bad idea.
>>
>> Disabling Thrift and Binary protocol should only avoid new connections,
>> but the one stablished and running should be attempted to finish.
>>
>> Any thoughts or comments?
>>
>> Thanks
>>
>> Javier.
>>
>>
>>
>
--
*Regards,*
*Anshu *
Re: Best approach to prepare to shutdown a cassandra node
Posted by Javier Canillas <ja...@gmail.com>.
That's what I thought.
Thanks!
2017-10-12 14:26 GMT-03:00 Hannu Kröger <hk...@gmail.com>:
> Hi,
>
> Drain should be enough. It stops accepting writes and after that
> cassandra can be safely shut down.
>
> Hannu
>
> On 12 October 2017 at 20:24:41, Javier Canillas (javier.canillas@gmail.com)
> wrote:
>
> Hello everyone,
>
> I have some time working with Cassandra, but every time I need to shutdown
> a node (for any reason like upgrading version or moving instance to another
> host) I see several errors on the client applications (yes, I'm using the
> official java driver).
>
> By the way, I'm starting C* as a stand-alone process
> <https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
> and C* version is 3.11.0.
>
> The way I have implemented the shutdown process is something like the
> following:
>
> *# Drain all information from commitlog into sstables*
>
> *bin/nodetool drain*
>
>
> *cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
> '{print $2}'`*
> *if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
> * echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
> cassandra)"*
> * kill $cassandra_pid*
>
> * echo -n "+ Checking it is down. "*
> * counter=10*
> * while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
> 2>&1 ]*
> * do*
> * echo -n ". "*
> * ((counter--))*
> * sleep 1s*
> * done*
> * echo ""*
> * if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
> * echo "+ Its down."*
> * else*
> * echo "- Killing Cassandra."*
> * kill -9 $cassandra_pid*
> * fi*
> *else*
> * echo "Care there was a problem finding Cassandra PID"*
> *fi*
>
> Should I add at the beginning the following lines?
>
> echo "shutdowing cassandra gracefully with: nodetool disable gossip"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
> echo "shutdowing cassandra gracefully with: nodetool disable binary
> protocol"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
> echo "shutdowing cassandra gracefully with: nodetool thrift"
> $CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
>
> The shutdown log is the following:
>
> *WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
> StorageService.java:321 - Stopping gossip by operator request*
> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
> Gossiper.java:1532 - Announcing shutdown*
> *INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
> StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
> jump to shutdown*
> *INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
> Server.java:176 - Stop listening for CQL clients*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
> StorageService.java:1442 - DRAINING: starting drain process*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
> HintsService.java:220 - Paused hints dispatch*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
> Gossiper.java:1532 - Announcing shutdown*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
> StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
> shutdown*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
> MessagingService.java:984 - Waiting for messaging service to quiesce*
> *INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
> 14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
> the accept() thread*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
> HintsService.java:220 - Paused hints dispatch*
> *INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
> StorageService.java:1442 - DRAINED*
>
> Disabling Gossip seemed a good idea, but watching the logs, it may use it
> to gracefully telling the other nodes he is going down, so I don't know if
> it's good or bad idea.
>
> Disabling Thrift and Binary protocol should only avoid new connections,
> but the one stablished and running should be attempted to finish.
>
> Any thoughts or comments?
>
> Thanks
>
> Javier.
>
>
>
Re: Best approach to prepare to shutdown a cassandra node
Posted by Hannu Kröger <hk...@gmail.com>.
Hi,
Drain should be enough. It stops accepting writes and after that cassandra
can be safely shut down.
Hannu
On 12 October 2017 at 20:24:41, Javier Canillas (javier.canillas@gmail.com)
wrote:
Hello everyone,
I have some time working with Cassandra, but every time I need to shutdown
a node (for any reason like upgrading version or moving instance to another
host) I see several errors on the client applications (yes, I'm using the
official java driver).
By the way, I'm starting C* as a stand-alone process
<https://docs.datastax.com/en/cassandra/3.0/cassandra/initialize/referenceStartCprocess.html?hl=start>,
and C* version is 3.11.0.
The way I have implemented the shutdown process is something like the
following:
*# Drain all information from commitlog into sstables*
*bin/nodetool drain*
*cassandra_pid=`ps -ef|grep "java.*apache-cassandra"|grep -v "grep"|awk
'{print $2}'`*
*if [ ! -z "$cassandra_pid" ] && [ "$cassandra_pid" -ne "1" ]; then*
* echo "Asking Cassandra to shutdown (nodetool drain doesn't stop
cassandra)"*
* kill $cassandra_pid*
* echo -n "+ Checking it is down. "*
* counter=10*
* while [ "$counter" -ne 0 -a ! kill -0 $cassandra_pid > /dev/null
2>&1 ]*
* do*
* echo -n ". "*
* ((counter--))*
* sleep 1s*
* done*
* echo ""*
* if ! kill -0 $cassandra_pid > /dev/null 2>&1; then*
* echo "+ Its down."*
* else*
* echo "- Killing Cassandra."*
* kill -9 $cassandra_pid*
* fi*
*else*
* echo "Care there was a problem finding Cassandra PID"*
*fi*
Should I add at the beginning the following lines?
echo "shutdowing cassandra gracefully with: nodetool disable gossip"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablegossip
echo "shutdowing cassandra gracefully with: nodetool disable binary
protocol"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablebinary
echo "shutdowing cassandra gracefully with: nodetool thrift"
$CASSANDRA_HOME/$CASSANDRA_APP/bin/nodetool disablethrift
The shutdown log is the following:
*WARN [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,343
StorageService.java:321 - Stopping gossip by operator request*
*INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,344
Gossiper.java:1532 - Announcing shutdown*
*INFO [RMI TCP Connection(10)-127.0.0.1] 2017-10-12 14:20:52,355
StorageService.java:2268 - Node /10.254.169.36 <http://10.254.169.36> state
jump to shutdown*
*INFO [RMI TCP Connection(12)-127.0.0.1] 2017-10-12 14:20:56,141
Server.java:176 - Stop listening for CQL clients*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,472
StorageService.java:1442 - DRAINING: starting drain process*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,474
HintsService.java:220 - Paused hints dispatch*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,477
Gossiper.java:1532 - Announcing shutdown*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:20:59,480
StorageService.java:2268 - Node /127.0.0.1 <http://127.0.0.1> state jump to
shutdown*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:01,483
MessagingService.java:984 - Waiting for messaging service to quiesce*
*INFO [ACCEPT-/192.168.6.174 <http://192.168.6.174>] 2017-10-12
14:21:01,485 MessagingService.java:1338 - MessagingService has terminated
the accept() thread*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,095
HintsService.java:220 - Paused hints dispatch*
*INFO [RMI TCP Connection(16)-127.0.0.1] 2017-10-12 14:21:02,111
StorageService.java:1442 - DRAINED*
Disabling Gossip seemed a good idea, but watching the logs, it may use it
to gracefully telling the other nodes he is going down, so I don't know if
it's good or bad idea.
Disabling Thrift and Binary protocol should only avoid new connections, but
the one stablished and running should be attempted to finish.
Any thoughts or comments?
Thanks
Javier.