You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Redmumba <re...@gmail.com> on 2014/04/02 00:24:11 UTC
Drop in node replacements.
Is it possible to have true "drop in" node replacements? For example, I
have a cluster of 51 Cassandra nodes, 17 in each data center. I had one
host go down on DC3, and when it came back up, it joined the ring, etc.,
but was not receiving any data. Even after multiple restarts and forcing a
repair on the entire fleet, it still holds maybe ~30MB on a cluster that is
absorbing ~1.2TB a day.
On top of that, I decided to see if I could recreate it--by taking down a
node, reprovisioning it, and then throwing it back in WITHOUT having it
"take over" the old node's tokens, it never seems to ever absorb any of the
old data after a full repair, and it never seems to start loading new data
(I now have 3 nodes using ~30MB).
Am I doing something wrong? I would imagine a repair on the entire cluster
(across 3 DCs) would force C* to put some copies onto the other node--but
this doesn't seem to be the case. What can I do?
Andrew
Re: Drop in node replacements.
Posted by Robert Coli <rc...@eventbrite.com>.
On Sat, Apr 5, 2014 at 5:10 PM, Anand Somani <me...@gmail.com> wrote:
> Have you tried nodetool rebuild for that node? I have seen that work when
> repair failed.
>
While rebuild may work in cases when repair doesn't, they do different
things and are not mutually substitutable.
"rebuild" is essentially bootstrap, streaming data from one source replica
per range.
"repair" syncs all replicas per range.
If you can successful "rebuild", it is true that afterwards your delta to
"repair" will be much smaller and potentially more likely to actually
succeed.
=Rob
Re: Drop in node replacements.
Posted by Anand Somani <me...@gmail.com>.
Have you tried nodetool rebuild for that node? I have seen that work when
repair failed.
On Wed, Apr 2, 2014 at 11:44 AM, Redmumba <re...@gmail.com> wrote:
> Cassandra 1.2.15, using commodity hardware.
>
>
> On Tue, Apr 1, 2014 at 6:37 PM, Robert Coli <rc...@eventbrite.com> wrote:
>
>> On Tue, Apr 1, 2014 at 3:24 PM, Redmumba <re...@gmail.com> wrote:
>>
>>> Is it possible to have true "drop in" node replacements? For example, I
>>> have a cluster of 51 Cassandra nodes, 17 in each data center. I had one
>>> host go down on DC3, and when it came back up, it joined the ring, etc.,
>>> but was not receiving any data. Even after multiple restarts and forcing a
>>> repair on the entire fleet, it still holds maybe ~30MB on a cluster that is
>>> absorbing ~1.2TB a day.
>>>
>>
>> What version of Cassandra? Real hardware/network or virtual?
>>
>> =Rob
>>
>>
>
Re: Drop in node replacements.
Posted by Redmumba <re...@gmail.com>.
Cassandra 1.2.15, using commodity hardware.
On Tue, Apr 1, 2014 at 6:37 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, Apr 1, 2014 at 3:24 PM, Redmumba <re...@gmail.com> wrote:
>
>> Is it possible to have true "drop in" node replacements? For example, I
>> have a cluster of 51 Cassandra nodes, 17 in each data center. I had one
>> host go down on DC3, and when it came back up, it joined the ring, etc.,
>> but was not receiving any data. Even after multiple restarts and forcing a
>> repair on the entire fleet, it still holds maybe ~30MB on a cluster that is
>> absorbing ~1.2TB a day.
>>
>
> What version of Cassandra? Real hardware/network or virtual?
>
> =Rob
>
>
Re: Drop in node replacements.
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, Apr 1, 2014 at 3:24 PM, Redmumba <re...@gmail.com> wrote:
> Is it possible to have true "drop in" node replacements? For example, I
> have a cluster of 51 Cassandra nodes, 17 in each data center. I had one
> host go down on DC3, and when it came back up, it joined the ring, etc.,
> but was not receiving any data. Even after multiple restarts and forcing a
> repair on the entire fleet, it still holds maybe ~30MB on a cluster that is
> absorbing ~1.2TB a day.
>
What version of Cassandra? Real hardware/network or virtual?
=Rob