You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Marcus Bointon <ma...@synchromedia.co.uk> on 2011/05/24 17:35:58 UTC
EC2 node adding trouble
Hi,
First time here. I'm having trouble adding a third node to an existing 2-node ring (successfully upgraded from 0.72) running cassandra 0.8rc1 (successfully upgraded from 0.72) on ubuntu on EC2.
Evidently the seed node is working as the second node is already talking to it, nodetool lists both nodes.
From the new node I can successfully telnet to ports 7000, 7199 and 9160 on the seed node (so it's not a connectivity issue - necessary ports are open on node firewalls and EC2 security groups).
All nodes have identical cassandra.yaml files, apart from the new node has auto-bootstrap set to true.
Despite all this, cassandra still says "No other nodes seen! Unable to bootstrap" when starting on the third node. Any ideas? How can I troubleshoot further?
Thanks,
Marcus
Re: EC2 node adding trouble
Posted by aaron morton <aa...@thelastpickle.com>.
This is the *most* useful page on the wiki http://wiki.apache.org/cassandra/Operations
Hope that helps.
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 27 May 2011, at 02:06, Marcus Bointon wrote:
> On 26 May 2011, at 15:21, Sasha Dolgy wrote:
>
>> Turn the node off, remove the node from the ring using nodetool and
>> removetoken .... i've found this to be the best problem-free way.
>> Maybe it's better now ...
>> http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html
>
> So I'd need to have at least replication=2 in order to do that safely? Your article makes it sound like draining/decommission doesn't work?
>
> Has anyone automated node addition/removal using chef or similar?
>
> Marcus
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 26 May 2011, at 15:21, Sasha Dolgy wrote:
> Turn the node off, remove the node from the ring using nodetool and
> removetoken .... i've found this to be the best problem-free way.
> Maybe it's better now ...
> http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html
So I'd need to have at least replication=2 in order to do that safely? Your article makes it sound like draining/decommission doesn't work?
Has anyone automated node addition/removal using chef or similar?
Marcus
Re: EC2 node adding trouble
Posted by Sasha Dolgy <sd...@gmail.com>.
On Thu, May 26, 2011 at 3:12 PM, Marcus Bointon
<ma...@synchromedia.co.uk> wrote:
> I'd like to make sure I've got the right sequence of operations for adding a node without downtime. If I'm going from 2 to 3 nodes:
>
> 1 Calculate new initial_token values using the python script
> 2 Change token values in existing nodes and restart them
> 3 Install/configure new node
> 4 Insert new node's token value
> 5 Set new node to auto-bootstrap
> 6 Start cassandra on new node
> 7 Wait for the ring to rebalance
>
> With token changes (using values from the python script), it's clear that all nodes will have some data moved. Does this mean that there's a possibility of overlap between regions if token changes are not absolutely simultaneous on all nodes? That sounds dangerous to me... Or shouldn't token values be changed on nodes containing data?
>
nodetool repair is good. when we add new nodes, we add a new one
without specifying the new token. after everything is up and healthy,
we determine new tokens and see if there is a need to renumber nodes.
if we do, we do one at a time and wait until the nodetool repair is
finished on one node before moving to another....
> Is there a corresponding sequence for removing nodes? I'm guessing draining is involved.
Turn the node off, remove the node from the ring using nodetool and
removetoken .... i've found this to be the best problem-free way.
Maybe it's better now ...
http://blog.sasha.dolgy.com/2011/03/apache-cassandra-nodetool.html
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 24 May 2011, at 23:58, Sameer Farooqui wrote:
> So, once you know what token each of the 3 nodes should have, shut down the first two nodes, change their tokens and add the correct token to the 3rd node (in the YAML file).
I'd like to make sure I've got the right sequence of operations for adding a node without downtime. If I'm going from 2 to 3 nodes:
1 Calculate new initial_token values using the python script
2 Change token values in existing nodes and restart them
3 Install/configure new node
4 Insert new node's token value
5 Set new node to auto-bootstrap
6 Start cassandra on new node
7 Wait for the ring to rebalance
With token changes (using values from the python script), it's clear that all nodes will have some data moved. Does this mean that there's a possibility of overlap between regions if token changes are not absolutely simultaneous on all nodes? That sounds dangerous to me... Or shouldn't token values be changed on nodes containing data?
Can cassandra nodes restart without downtime?
I'm looking at http://wiki.apache.org/cassandra/MultinodeCluster but as it says it's deliberately simplistic.
Is there a corresponding sequence for removing nodes? I'm guessing draining is involved.
Marcus
Re: EC2 node adding trouble
Posted by aaron morton <aa...@thelastpickle.com>.
This ticket may be just the ticket :)
https://issues.apache.org/jira/browse/CASSANDRA-2452
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 27 May 2011, at 01:16, Sasha Dolgy wrote:
> As an aside, you can also use that command to pull meta-data about
> instances in AWS. I have implemented this to maintain a list of seed
> nodes. This way, when a new instance is brought online, the default
> cassandra.yaml is `enhanced` to contain a dynamic list of valid seeds,
> proper hostname and a few other bits of useful information.
>
> Finally, if you aren't using a single security group for all of your
> cassandra instances, maybe this may be of help to you. When we add
> new nodes to our ring, we add them to a single cassandra security
> group. No messing about with security groups per instance...
>
> -sd
>
> On Thu, May 26, 2011 at 2:36 PM, Marcus Bointon
> <ma...@synchromedia.co.uk> wrote:
>> Thanks for all your helpful suggestions - I've now got it working. It was down to a combination of things.
>>
>> 1. A missing rule in a security group
>> 2. A missing DNS name for the new node, so its default name was defaulting to localhost
>> 3. Google DNS caching the failed DNS lookup for the full duration of the SOA's TTL
>>
>> In order to avoid the whole problem with assigning IPs using the internal/external trick and using up elastic IPs, I found this service which I'd not seen before: http://www.ducea.com/2009/06/01/howto-update-dns-hostnames-automatically-for-your-amazon-ec2-instances/
>>
>> This means you can reliably set (and reset as necessary) a listen address with this command:
>>
>> sed -i "s/^listen_address:.*/listen_address: `curl http://169.254.169.254/latest/meta-data/local-ipv4`/" /etc/cassandra/cassandra.yaml
>>
>> It's not quite as good as having a true dynamic hostname, but at least you can drop it in a startup script and forget it.
>>
>> Marcus
Re: EC2 node adding trouble
Posted by Sasha Dolgy <sd...@gmail.com>.
As an aside, you can also use that command to pull meta-data about
instances in AWS. I have implemented this to maintain a list of seed
nodes. This way, when a new instance is brought online, the default
cassandra.yaml is `enhanced` to contain a dynamic list of valid seeds,
proper hostname and a few other bits of useful information.
Finally, if you aren't using a single security group for all of your
cassandra instances, maybe this may be of help to you. When we add
new nodes to our ring, we add them to a single cassandra security
group. No messing about with security groups per instance...
-sd
On Thu, May 26, 2011 at 2:36 PM, Marcus Bointon
<ma...@synchromedia.co.uk> wrote:
> Thanks for all your helpful suggestions - I've now got it working. It was down to a combination of things.
>
> 1. A missing rule in a security group
> 2. A missing DNS name for the new node, so its default name was defaulting to localhost
> 3. Google DNS caching the failed DNS lookup for the full duration of the SOA's TTL
>
> In order to avoid the whole problem with assigning IPs using the internal/external trick and using up elastic IPs, I found this service which I'd not seen before: http://www.ducea.com/2009/06/01/howto-update-dns-hostnames-automatically-for-your-amazon-ec2-instances/
>
> This means you can reliably set (and reset as necessary) a listen address with this command:
>
> sed -i "s/^listen_address:.*/listen_address: `curl http://169.254.169.254/latest/meta-data/local-ipv4`/" /etc/cassandra/cassandra.yaml
>
> It's not quite as good as having a true dynamic hostname, but at least you can drop it in a startup script and forget it.
>
> Marcus
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
Thanks for all your helpful suggestions - I've now got it working. It was down to a combination of things.
1. A missing rule in a security group
2. A missing DNS name for the new node, so its default name was defaulting to localhost
3. Google DNS caching the failed DNS lookup for the full duration of the SOA's TTL
In order to avoid the whole problem with assigning IPs using the internal/external trick and using up elastic IPs, I found this service which I'd not seen before: http://www.ducea.com/2009/06/01/howto-update-dns-hostnames-automatically-for-your-amazon-ec2-instances/
This means you can reliably set (and reset as necessary) a listen address with this command:
sed -i "s/^listen_address:.*/listen_address: `curl http://169.254.169.254/latest/meta-data/local-ipv4`/" /etc/cassandra/cassandra.yaml
It's not quite as good as having a true dynamic hostname, but at least you can drop it in a startup script and forget it.
Marcus
Re: EC2 node adding trouble
Posted by mcasandra <mo...@gmail.com>.
Can you post the output of "netstat -anp|grep "LISTEN"|grep java" from all
the 3 nodes?
Also compare seconds nodes yaml with new nodes yaml and see what diff. you
find, if any.
Another thing try telnet tests from seed node to the new node.
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/EC2-node-adding-trouble-tp6399102p6403602.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 24 May 2011, at 23:58, Sameer Farooqui wrote:
> Even with AutoBootstrap it is recommended that you always specify the InitialToken on the new node because the picking of an initial token will almost certainly result in an unbalanced ring.
>
> Right now, I'm afraid that if you simply copied the YAML file from one of the two nodes to the 3rd node, then the 3rd node has an incorrect initial_token setting (it may be conflicting with node 1 or 2's range).
>
> Some info about how to choose tokens:
> http://wiki.apache.org/cassandra/Operations#Token_selection
> http://journal.paul.querna.org/articles/2010/09/24/cassandra-token-selection/
>
> So, once you know what token each of the 3 nodes should have, shut down the first two nodes, change their tokens and add the correct token to the 3rd node (in the YAML file).
At the moment I'm not bothered about balance - just having it work at all would be good! initial_token is currently blank (i.e. self-defining according to the docs) on all nodes - I can see why that might cause balance issues, but I don't see that it would prevent it working as there should not be a token clash. If there was a token clash, wouldn't it throw an error saying so?
That second article, while wonderfully practical and informative, suggests that cassandra is really not suited to rapid scaling up or down - i.e. adding or removing nodes is neither quick nor painless, e.g. scaling from 5 to 20 nodes and back is something that might happen over weeks rather than minutes. Maybe cassandra just isn't the right tool for the job in this case?
Marcus
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 26 May 2011, at 00:17, aaron morton wrote:
> I've seen discussion of using the EIP but I do not have direct experience.
The idea is not to use the external IP, but the external DNS name because of this very useful trick (please excuse me if you already know this!):
Say the DNS name of an elastic IP assigned to an instance is ec2-50-18-223-109.compute-1.amazonaws.com, then from outside EC2:
#host ec2-50-18-223-109.compute-1.amazonaws.com
ec2-50-18-223-109.compute-1.amazonaws.com has address 50.18.223.109
But from inside EC2:
#host ec2-50-18-223-109.compute-1.amazonaws.com
ec2-50-18-223-109.compute-1.amazonaws.com has address 10.126.13.22
If you suspend and resume an instance, its internal IP will change, but the external will not (if it's assigned the same elastic IP), but if you use the external name, you'll get consistent behaviour whatever happens. This is extremely useful!
Of course it would be extremely useful if we could get behaviour like this without having to assign an elastic IP, as this is a waste of IPs otherwise.
Marcus
Re: EC2 node adding trouble
Posted by Sasha Dolgy <sd...@gmail.com>.
All of the nodes are 0.8.x and no longer 0.7.2 ? It is peculiar. As
another person noted, maybe reviewing the results of netstat -an on
each node could help. I've had issues in the past with JMX --
thinking I had configured it for IP x.x.x.x and went crazy only to see
that it was never configured for x.x.x.x despite the configurations.
another thing to try look at is verifying that DNS is working properly
on the new node. is it a stock instance or has the hostname been
changed, etc. that also causes issues ...
On Thu, May 26, 2011 at 12:17 AM, aaron morton <aa...@thelastpickle.com> wrote:
> I've seen discussion of using the EIP but I do not have direct experience.
> Others may be able to provide more help.
> Previous discussion
> http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Does-anybody-have-experience-with-running-Cassandra-in-Amazon-s-Virtual-Private-Cloud-VPC-td6335709.html
Re: EC2 node adding trouble
Posted by aaron morton <aa...@thelastpickle.com>.
I've seen discussion of using the EIP but I do not have direct experience.
Others may be able to provide more help.
Previous discussion
http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Does-anybody-have-experience-with-running-Cassandra-in-Amazon-s-Virtual-Private-Cloud-VPC-td6335709.html
cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 25 May 2011, at 18:49, Marcus Bointon wrote:
> On 25 May 2011, at 02:10, aaron morton wrote:
>
>> Check the listen_address and rpc_address in the yaml file for each node. I think they are normally set to the private and public respectively.
>
> On all modes, listen_address and rpc_address are blank and 0.0.0.0 respectively, the seed node address uses an EC2 external elastic IP's DNS name so that it resolves correctly to the seed's internal IP even if it changes. Those settings work fine for my existing pair - starting and stopping the instances (resulting in internal IP changes) doesn't break anything. As far as I can see, setting fixed IPs in here is a recipe for management nightmares on EC2.
>
> Sound sane?
>
> Marcus
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 25 May 2011, at 02:10, aaron morton wrote:
> Check the listen_address and rpc_address in the yaml file for each node. I think they are normally set to the private and public respectively.
On all modes, listen_address and rpc_address are blank and 0.0.0.0 respectively, the seed node address uses an EC2 external elastic IP's DNS name so that it resolves correctly to the seed's internal IP even if it changes. Those settings work fine for my existing pair - starting and stopping the instances (resulting in internal IP changes) doesn't break anything. As far as I can see, setting fixed IPs in here is a recipe for management nightmares on EC2.
Sound sane?
Marcus
Re: EC2 node adding trouble
Posted by aaron morton <aa...@thelastpickle.com>.
Check the listen_address and rpc_address in the yaml file for each node. I think they are normally set to the private and public respectively.
This may make your live easier
http://www.datastax.com/dev/blog/setting-up-a-cassandra-cluster-with-the-datastax-ami
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 25 May 2011, at 09:58, Sameer Farooqui wrote:
> Even with AutoBootstrap it is recommended that you always specify the InitialToken on the new node because the picking of an initial token will almost certainly result in an unbalanced ring.
>
> Right now, I'm afraid that if you simply copied the YAML file from one of the two nodes to the 3rd node, then the 3rd node has an incorrect initial_token setting (it may be conflicting with node 1 or 2's range).
>
> Some info about how to choose tokens:
> http://wiki.apache.org/cassandra/Operations#Token_selection
> http://journal.paul.querna.org/articles/2010/09/24/cassandra-token-selection/
>
> So, once you know what token each of the 3 nodes should have, shut down the first two nodes, change their tokens and add the correct token to the 3rd node (in the YAML file).
>
> If this doesn't fix it, start to look at the Cassandra log file to troubleshoot. It's located usually at /var/log/cassandra and is by default only logging at INFO level. You can make it more granular by changing the logging level in the file <cassandra-home>/conf/log4j-server.properties and change the line with INFO in it to:
> log4j.rootLogger=DEBUG,stdout,R
>
> Note that this will slow things down performance-wise in the cluster since every I/O is logged. But it really helps with learning how Cassandra works and with troubleshooting.
>
> research links:
> http://wiki.apache.org/cassandra/FAQ#existing_data_when_adding_new_nodes
>
>
>
> On Tue, May 24, 2011 at 12:17 PM, Marcus Bointon <ma...@synchromedia.co.uk> wrote:
> On 24 May 2011, at 19:33, Sameer Farooqui wrote:
>
> > What region and availability zones are the different nodes in? Are you using EC2 Snitch? Did you set up the cluster using the Datastax AMI?
>
> The two existing ones are in us-east-1c and us-east-1d, the new one is in us-east-1c, so all same region, different AZs. I'm using the Alestic Ubuntu AMIs. I've not heard of either EC2 snitch or Datastax. Are they particularly suited to cassandra?
>
> Marcus
>
Re: EC2 node adding trouble
Posted by Sameer Farooqui <ca...@gmail.com>.
Even with AutoBootstrap it is recommended that you always specify
the InitialToken on the new node because the picking of an initial token
will almost certainly result in an unbalanced ring.
Right now, I'm afraid that if you simply copied the YAML file from one of
the two nodes to the 3rd node, then the 3rd node has an incorrect
initial_token setting (it may be conflicting with node 1 or 2's range).
Some info about how to choose tokens:
http://wiki.apache.org/cassandra/Operations#Token_selection
<http://wiki.apache.org/cassandra/Operations#Token_selection>
http://journal.paul.querna.org/articles/2010/09/24/cassandra-token-selection/
So, once you know what token each of the 3 nodes should have, shut down the
first two nodes, change their tokens and add the correct token to the 3rd
node (in the YAML file).
If this doesn't fix it, start to look at the Cassandra log file to
troubleshoot. It's located usually at /var/log/cassandra and is by default
only logging at INFO level. You can make it more granular by changing the
logging level in the file <cassandra-home>/conf/log4j-server.properties and
change the line with INFO in it to:
log4j.rootLogger=DEBUG,stdout,R
Note that this will slow things down performance-wise in the cluster since
every I/O is logged. But it really helps with learning how Cassandra works
and with troubleshooting.
research links:
http://wiki.apache.org/cassandra/FAQ#existing_data_when_adding_new_nodes
On Tue, May 24, 2011 at 12:17 PM, Marcus Bointon
<ma...@synchromedia.co.uk>wrote:
> On 24 May 2011, at 19:33, Sameer Farooqui wrote:
>
> > What region and availability zones are the different nodes in? Are you
> using EC2 Snitch? Did you set up the cluster using the Datastax AMI?
>
> The two existing ones are in us-east-1c and us-east-1d, the new one is in
> us-east-1c, so all same region, different AZs. I'm using the Alestic Ubuntu
> AMIs. I've not heard of either EC2 snitch or Datastax. Are they particularly
> suited to cassandra?
>
> Marcus
Re: EC2 node adding trouble
Posted by Marcus Bointon <ma...@synchromedia.co.uk>.
On 24 May 2011, at 19:33, Sameer Farooqui wrote:
> What region and availability zones are the different nodes in? Are you using EC2 Snitch? Did you set up the cluster using the Datastax AMI?
The two existing ones are in us-east-1c and us-east-1d, the new one is in us-east-1c, so all same region, different AZs. I'm using the Alestic Ubuntu AMIs. I've not heard of either EC2 snitch or Datastax. Are they particularly suited to cassandra?
Marcus
Re: EC2 node adding trouble
Posted by Sameer Farooqui <ca...@gmail.com>.
What region and availability zones are the different nodes in? Are you using
EC2 Snitch? Did you set up the cluster using the Datastax AMI?
- Sameer
On Tue, May 24, 2011 at 8:35 AM, Marcus Bointon
<ma...@synchromedia.co.uk>wrote:
> Hi,
>
> First time here. I'm having trouble adding a third node to an existing
> 2-node ring (successfully upgraded from 0.72) running cassandra 0.8rc1
> (successfully upgraded from 0.72) on ubuntu on EC2.
>
> Evidently the seed node is working as the second node is already talking to
> it, nodetool lists both nodes.
> From the new node I can successfully telnet to ports 7000, 7199 and 9160 on
> the seed node (so it's not a connectivity issue - necessary ports are open
> on node firewalls and EC2 security groups).
> All nodes have identical cassandra.yaml files, apart from the new node has
> auto-bootstrap set to true.
>
> Despite all this, cassandra still says "No other nodes seen! Unable to
> bootstrap" when starting on the third node. Any ideas? How can I
> troubleshoot further?
>
> Thanks,
>
> Marcus