Restarting servers

[Jason Baker <ja...@apture.com>]

So restarting cassandra servers has a tendency to cause a lot of exceptions
like "MaximumRetryException: Retried 6 times. Last failure was
UnavailableException()" and "TApplicationException: Internal error
processing batch_mutate" (using pycassa).  If I restart the servers too
quickly, I get "all servers unavailable".  So two questions:

1.  Is there anything I can do to prevent MaximumRetryExceptions and
TApplicationExceptions, or is this just a case of needing better exception
handling?
2.  Are there any rules of thumb regarding how much time I should allow
between server restarts?

Re: Restarting servers

[Jeremiah Jordan <je...@morningstar.com>]

You need to wait for the servers to be up again before restarting the 
next one.  nodetool ring on one of the servers you aren't restarting 
will tell you when it is back up.  You can also watch for "Starting up 
server gossip" in the log file to know when it is starting to join the 
cluster again.

On 08/12/2011 01:59 PM, Jason Baker wrote:
> So restarting cassandra servers has a tendency to cause a lot of 
> exceptions like "MaximumRetryException: Retried 6 times. Last failure 
> was UnavailableException()" and "TApplicationException: Internal error 
> processing batch_mutate" (using pycassa).  If I restart the servers 
> too quickly, I get "all servers unavailable".  So two questions:
>
> 1.  Is there anything I can do to prevent MaximumRetryExceptions and 
> TApplicationExceptions, or is this just a case of needing better 
> exception handling?
> 2.  Are there any rules of thumb regarding how much time I should 
> allow between server restarts?