You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nicolas Lalevée <ni...@hibnet.org> on 2012/06/08 14:10:40 UTC
Dead node still being pinged
I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
To move all the data without interruption of service, I added one new node at a time. And then I removed one by one the old machines via a "remove token".
Everything was working fine. Until there was an expected load on our cluster, the machine started to swap and become unresponsive. We fixed the unexpected load and the three new machines were restarted. After that the new cassandra machines were stating that some old token were not assigned, namely from data-2 and data-4. To fix this I issued again some "remove token" commands.
Everything seems to be back to normal, but on the network I still see some packet from the new cluster to the old machines. On the port 7000.
How I can tell cassandra to completely forget about the old machines ?
Nicolas
Re: Dead node still being pinged
Posted by Nicolas Lalevée <ni...@hibnet.org>.
Le 8 juin 2012 à 20:50, aaron morton a écrit :
> Are the old machines listed in the seed list on the new ones ?
No they don't.
The first of my old node was, when I was "migrating". But not anymore.
Nicolas
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 9/06/2012, at 12:10 AM, Nicolas Lalevée wrote:
>
>> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
>> To move all the data without interruption of service, I added one new node at a time. And then I removed one by one the old machines via a "remove token".
>>
>> Everything was working fine. Until there was an expected load on our cluster, the machine started to swap and become unresponsive. We fixed the unexpected load and the three new machines were restarted. After that the new cassandra machines were stating that some old token were not assigned, namely from data-2 and data-4. To fix this I issued again some "remove token" commands.
>>
>> Everything seems to be back to normal, but on the network I still see some packet from the new cluster to the old machines. On the port 7000.
>> How I can tell cassandra to completely forget about the old machines ?
>>
>> Nicolas
>>
>
Re: Dead node still being pinged
Posted by aaron morton <aa...@thelastpickle.com>.
Are the old machines listed in the seed list on the new ones ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 9/06/2012, at 12:10 AM, Nicolas Lalevée wrote:
> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
> To move all the data without interruption of service, I added one new node at a time. And then I removed one by one the old machines via a "remove token".
>
> Everything was working fine. Until there was an expected load on our cluster, the machine started to swap and become unresponsive. We fixed the unexpected load and the three new machines were restarted. After that the new cassandra machines were stating that some old token were not assigned, namely from data-2 and data-4. To fix this I issued again some "remove token" commands.
>
> Everything seems to be back to normal, but on the network I still see some packet from the new cluster to the old machines. On the port 7000.
> How I can tell cassandra to completely forget about the old machines ?
>
> Nicolas
>
Re: Dead node still being pinged
Posted by Nicolas Lalevée <ni...@hibnet.org>.
Le 8 juin 2012 à 15:17, Samuel CARRIERE a écrit :
> What does nodetool ring says ? (Ask every node)
currently, each of new node see only the tokens of the new nodes.
> Have you checked that the list of seeds in every yaml is correct ?
yes, it is correct, every of my new node point to the first of my new node
> What version of cassandra are you using ?
Sorry I should have wrote this in my first mail.
I use the 1.0.9
Nicolas
>
> Samuel
>
>
>
> Nicolas Lalevée <ni...@hibnet.org>
> 08/06/2012 14:10
> Veuillez répondre à
> user@cassandra.apache.org
>
> A
> user@cassandra.apache.org
> cc
> Objet
> Dead node still being pinged
>
>
>
>
>
> I had a configuration where I had 4 nodes, data-1,4. We then bought 3 bigger machines, data-5,7. And we moved all data from data-1,4 to data-5,7.
> To move all the data without interruption of service, I added one new node at a time. And then I removed one by one the old machines via a "remove token".
>
> Everything was working fine. Until there was an expected load on our cluster, the machine started to swap and become unresponsive. We fixed the unexpected load and the three new machines were restarted. After that the new cassandra machines were stating that some old token were not assigned, namely from data-2 and data-4. To fix this I issued again some "remove token" commands.
>
> Everything seems to be back to normal, but on the network I still see some packet from the new cluster to the old machines. On the port 7000.
> How I can tell cassandra to completely forget about the old machines ?
>
> Nicolas
>
>
RE Dead node still being pinged
Posted by Samuel CARRIERE <sa...@urssaf.fr>.
Hi Nicolas,
What does nodetool ring says ? (Ask every node)
Have you checked that the list of seeds in every yaml is correct ?
What version of cassandra are you using ?
Samuel
Nicolas Lalevée <ni...@hibnet.org>
08/06/2012 14:10
Veuillez répondre à
user@cassandra.apache.org
A
user@cassandra.apache.org
cc
Objet
Dead node still being pinged
I had a configuration where I had 4 nodes, data-1,4. We then bought 3
bigger machines, data-5,7. And we moved all data from data-1,4 to
data-5,7.
To move all the data without interruption of service, I added one new node
at a time. And then I removed one by one the old machines via a "remove
token".
Everything was working fine. Until there was an expected load on our
cluster, the machine started to swap and become unresponsive. We fixed the
unexpected load and the three new machines were restarted. After that the
new cassandra machines were stating that some old token were not assigned,
namely from data-2 and data-4. To fix this I issued again some "remove
token" commands.
Everything seems to be back to normal, but on the network I still see some
packet from the new cluster to the old machines. On the port 7000.
How I can tell cassandra to completely forget about the old machines ?
Nicolas