You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Anthony Grasso <an...@gmail.com> on 2013/08/26 07:32:30 UTC
Cluster Management
Hi Cassandra Users,
Before I go ahead and create my own solution... are there any tools that
exist to help with the management of a Cassandra cluster?
For example, if I want to make some changes to the configuration file that
resides on each node, is there a tool that will propagate the change to
each node?
Another example is if I want to have a rolling repair (nodetool repair -pr)
and clean up running on my cluster, is there a tool that will help
manage/configure that?
Any feedback would be greatly appreciated.
Thanks,
Anthony
Re: Cluster Management
Posted by Anthony Grasso <an...@gmail.com>.
Thanks Nate! We will look into this one to see if we can use it.
Regards,
Anthony
On Tue, Aug 27, 2013 at 12:22 AM, Nate McCall <na...@thelastpickle.com>wrote:
>
>> For example, if I want to make some changes to the configuration file
>> that resides on each node, is there a tool that will propagate the change
>> to each node?
>>
>> You may also want to take a look at Priam from the Netflix folks:
> https://github.com/Netflix/Priam
>
> Assumes AWS (though some of this is becoming more plug-able).
>
Re: Cluster Management
Posted by Nate McCall <na...@thelastpickle.com>.
>
>
> For example, if I want to make some changes to the configuration file that
> resides on each node, is there a tool that will propagate the change to
> each node?
>
> You may also want to take a look at Priam from the Netflix folks:
https://github.com/Netflix/Priam
Assumes AWS (though some of this is becoming more plug-able).
Re: Cluster Management
Posted by Jonathan Haddad <jo...@jonhaddad.com>.
An alternative to cssh is fabric. It's very flexible in that you can
automate almost any repetitive task that you'd send to machines in a
cluster, and it's written in python, meaning if you're in AWS you can mix
it with boto to automate pretty much anything you want.
On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso <an...@gmail.com>wrote:
> Hi Particia,
>
> Thank you for the feedback. It has been helpful.
>
>
> On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla <gorla.patricia@gmail.com
> > wrote:
>
>> Anthony,
>>
>> We use a number of tools to manage our Cassandra cluster.
>>
>> * Datastax OpsCenter [0] for at a glance information, and trending
>> statistics. You can also run operations through here, though I prefer
>> to use nodetool for any mutative operation.
>> * nodetool for ad hoc status checks, and day-to-day node management.
>> * puppet for setup and initialization
>>
>> > For example, if I want to make some changes to the configuration file
>> that resides on each node, is there a tool that will propagate the change
>> to each node?
>>
>> For this, we use puppet to manage any changes to the configurations
>> (which are stored in git). We initially had Cassandra auto-restart
>> when the configuration changed, but you might not want the node to
>> automatically join a cluster, so we turned this off.
>>
>
> Puppet was the first thing that came to mind for us as well. In addition,
> we had the same thought about auto-restarting nodes when the configuration
> is changed. If a configuration on all the nodes is changed, we would want
> to restart one node at a time and wait for it to rejoin before restarting
> the next one. I am assuming in a case like this, you then manually perform
> the restart operation for each node?
>
>
>>
>> > Another example is if I want to have a rolling repair (nodetool repair
>> -pr) and clean up running on my cluster, is there a tool that will help
>> manage/configure that?
>>
>> Multiple commands to the cluster are sent via clusterssh [1] (cssh for
>> OS X). I can easily choose which nodes to control, and run those in
>> sync. For any rolling procedures, we send commands one at a time,
>> though we've considered sending some of these tasks to cron.
>>
>
> Thanks again for the tip! This is quite interesting; it may help to solve
> our immediate problem for now.
>
> Regards,
> Anthony
>
>
>>
>> Hope this helps.
>>
>> Cheers,
>> Patricia
>>
>>
>> [0] http://planetcassandra.org/Download/DataStaxCommunityEdition
>> [1] http://sourceforge.net/projects/clusterssh/
>>
>
>
--
Jon Haddad
http://www.rustyrazorblade.com
skype: rustyrazorblade
Re: Cluster Management
Posted by Robert Coli <rc...@eventbrite.com>.
On Thu, Aug 29, 2013 at 4:25 PM, Anthony Grasso <an...@gmail.com>wrote:
> Puppet was the first thing that came to mind for us as well. In addition,
> we had the same thought about auto-restarting nodes when the configuration
> is changed. If a configuration on all the nodes is changed, we would want
> to restart one node at a time and wait for it to rejoin before restarting
> the next one. I am assuming in a case like this, you then manually perform
> the restart operation for each node?
>
FYI, potentially relevant :
https://issues.apache.org/jira/browse/CASSANDRA-2356
=Rob
Re: Cluster Management
Posted by Anthony Grasso <an...@gmail.com>.
Hi Particia,
Thank you for the feedback. It has been helpful.
On Tue, Aug 27, 2013 at 12:02 AM, Patricia Gorla
<go...@gmail.com>wrote:
> Anthony,
>
> We use a number of tools to manage our Cassandra cluster.
>
> * Datastax OpsCenter [0] for at a glance information, and trending
> statistics. You can also run operations through here, though I prefer
> to use nodetool for any mutative operation.
> * nodetool for ad hoc status checks, and day-to-day node management.
> * puppet for setup and initialization
>
> > For example, if I want to make some changes to the configuration file
> that resides on each node, is there a tool that will propagate the change
> to each node?
>
> For this, we use puppet to manage any changes to the configurations
> (which are stored in git). We initially had Cassandra auto-restart
> when the configuration changed, but you might not want the node to
> automatically join a cluster, so we turned this off.
>
Puppet was the first thing that came to mind for us as well. In addition,
we had the same thought about auto-restarting nodes when the configuration
is changed. If a configuration on all the nodes is changed, we would want
to restart one node at a time and wait for it to rejoin before restarting
the next one. I am assuming in a case like this, you then manually perform
the restart operation for each node?
>
> > Another example is if I want to have a rolling repair (nodetool repair
> -pr) and clean up running on my cluster, is there a tool that will help
> manage/configure that?
>
> Multiple commands to the cluster are sent via clusterssh [1] (cssh for
> OS X). I can easily choose which nodes to control, and run those in
> sync. For any rolling procedures, we send commands one at a time,
> though we've considered sending some of these tasks to cron.
>
Thanks again for the tip! This is quite interesting; it may help to solve
our immediate problem for now.
Regards,
Anthony
>
> Hope this helps.
>
> Cheers,
> Patricia
>
>
> [0] http://planetcassandra.org/Download/DataStaxCommunityEdition
> [1] http://sourceforge.net/projects/clusterssh/
>
Re: Cluster Management
Posted by Patricia Gorla <go...@gmail.com>.
Anthony,
We use a number of tools to manage our Cassandra cluster.
* Datastax OpsCenter [0] for at a glance information, and trending
statistics. You can also run operations through here, though I prefer
to use nodetool for any mutative operation.
* nodetool for ad hoc status checks, and day-to-day node management.
* puppet for setup and initialization
> For example, if I want to make some changes to the configuration file that resides on each node, is there a tool that will propagate the change to each node?
For this, we use puppet to manage any changes to the configurations
(which are stored in git). We initially had Cassandra auto-restart
when the configuration changed, but you might not want the node to
automatically join a cluster, so we turned this off.
> Another example is if I want to have a rolling repair (nodetool repair -pr) and clean up running on my cluster, is there a tool that will help manage/configure that?
Multiple commands to the cluster are sent via clusterssh [1] (cssh for
OS X). I can easily choose which nodes to control, and run those in
sync. For any rolling procedures, we send commands one at a time,
though we've considered sending some of these tasks to cron.
Hope this helps.
Cheers,
Patricia
[0] http://planetcassandra.org/Download/DataStaxCommunityEdition
[1] http://sourceforge.net/projects/clusterssh/