You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Analia Lorenzatto <an...@gmail.com> on 2015/05/12 16:39:25 UTC
Unexpected behavior after adding successffully a new node
Hello guys,
I have a cluster 2.1.0-2 comprised of 3 nodes. The replication factor=2.
We successfully added the third node last week. After that, We ran clean
ups on one node at that time. Then we ran repairs on all the nodes, and
finally compactions on all the CFs.
Last night, I noticed the cluster started behaving in a weird way. The
last node (successfully added last week) were being reported up and down
all the time. I could see a lot of messages like this on logs:
WARN [SharedPool-Worker-33] 2015-05-11 21:31:45,125
AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
Thread[SharedPool-Worker-33,5,main]: {}
java.lang.RuntimeException: java.io.FileNotFoundException:
/mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
(No such file or directory)
At the same time the consumption of heap used was on the top, up to the
point the rest of the cluster saw this node as down. After that, I just
restarted the cassandra service with no problems on that node.
Now, I can see the three nodes on the cluster Up and Normal, but this last
node (which was rebooted) does not have data. But it has all the structure
of cassandra data.
I can query against the new node and I get the same result as if do the
query against the others nodes. But, on this new node I do not have any
SStables:
root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.0.0.a 390.28 GB 256 66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
UN 10.0.0.b 382.36 GB 256 68.3%
19492c26-4458-4a0b-af04-72e0aab6598e 1c
UN 10.0.0.c 40.61 MB 256 64.9%
b8da952c-24b3-444a-a34e-7a1804eee6e6 1c
What do you recommend to do? Leave this as if, remove it and try to join
this or a new one?
Thanks in advance!!
--
Saludos / Regards.
Analía Lorenzatto.
“It's possible to commit no errors and still lose. That is not weakness.
That is life". By Captain Jean-Luc Picard.
Re: Unexpected behavior after adding successffully a new node
Posted by Analia Lorenzatto <an...@gmail.com>.
Just in case I want to clarify that after bootstrapping the third node, it
got data and seemed to be working fine. But it was the last night when
the cluster started behaving in a weird way. The last node (successfully
added last week) were being reported up and down all the time. After
restarting cassandra service, the node got empty but reported as Up and
Normal.
Also, the amount of data reported by nodetool on the other nodes is much
more than the amount already used in the device, is that normal?
root@10.0.0.a:~# df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/xvdb1 745G 163G 583G 22% /mnt/cassandra
root@:10.0.0.b~$ df -h /mnt
Filesystem Size Used Avail Use% Mounted on
/dev/xvdb1 745G 196G 550G 27% /mnt/cassandra
# nodetool status
Datacenter: us-east
===================
Status=Up/Down
|/ State=Normal/Leaving/Joining/Moving
-- Address Load Tokens Owns (effective) Host ID
Rack
UN 10.0.0.a 391.34 GB 256 66.7%
eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
UN 10.0.0.b 383.38 GB 256 68.3%
19492c26-4458-4a0b-af04-72e0aab6598e 1c
Thanks in advance!
On Tue, May 12, 2015 at 2:35 PM, Robert Coli <rc...@eventbrite.com> wrote:
> On Tue, May 12, 2015 at 9:59 AM, arun sirimalla <ar...@gmail.com>
> wrote:
>
>> Try running repair on node 3.
>>
>
> Mostly disagree. If a node is empty after a bootstrap, remove it and
> re-bootstrap it.
>
> =Rob
>
>
--
Saludos / Regards.
Analía Lorenzatto.
“It's possible to commit no errors and still lose. That is not weakness.
That is life". By Captain Jean-Luc Picard.
Re: Unexpected behavior after adding successffully a new node
Posted by Robert Coli <rc...@eventbrite.com>.
On Tue, May 12, 2015 at 9:59 AM, arun sirimalla <ar...@gmail.com> wrote:
> Try running repair on node 3.
>
Mostly disagree. If a node is empty after a bootstrap, remove it and
re-bootstrap it.
=Rob
Re: Unexpected behavior after adding successffully a new node
Posted by arun sirimalla <ar...@gmail.com>.
Analia,
Try running repair on node 3.
On Tue, May 12, 2015 at 7:39 AM, Analia Lorenzatto <
analialorenzatto@gmail.com> wrote:
> Hello guys,
>
>
> I have a cluster 2.1.0-2 comprised of 3 nodes. The replication factor=2.
> We successfully added the third node last week. After that, We ran clean
> ups on one node at that time. Then we ran repairs on all the nodes, and
> finally compactions on all the CFs.
>
> Last night, I noticed the cluster started behaving in a weird way. The
> last node (successfully added last week) were being reported up and down
> all the time. I could see a lot of messages like this on logs:
>
> WARN [SharedPool-Worker-33] 2015-05-11 21:31:45,125
> AbstractTracingAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[SharedPool-Worker-33,5,main]: {}
> java.lang.RuntimeException: java.io.FileNotFoundException:
> /mnt/cassandra/data/matchings-85b4929048e211e4a949a3ed319cbedc/matchings-ka-3914-Data.db
> (No such file or directory)
>
> At the same time the consumption of heap used was on the top, up to the
> point the rest of the cluster saw this node as down. After that, I just
> restarted the cassandra service with no problems on that node.
>
> Now, I can see the three nodes on the cluster Up and Normal, but this last
> node (which was rebooted) does not have data. But it has all the structure
> of cassandra data.
>
> I can query against the new node and I get the same result as if do the
> query against the others nodes. But, on this new node I do not have any
> SStables:
>
>
>
> root@prd-rtbkit-cassandra-03:/var/log/cassandra# nodetool status
> Datacenter: us-east
> ===================
> Status=Up/Down
> |/ State=Normal/Leaving/Joining/Moving
> -- Address Load Tokens Owns (effective) Host ID
> Rack
> UN 10.0.0.a 390.28 GB 256 66.7%
> eed9e9f5-f279-4b2f-b521-c056cbf65b52 1c
> UN 10.0.0.b 382.36 GB 256 68.3%
> 19492c26-4458-4a0b-af04-72e0aab6598e 1c
> UN 10.0.0.c 40.61 MB 256 64.9%
> b8da952c-24b3-444a-a34e-7a1804eee6e6 1c
>
> What do you recommend to do? Leave this as if, remove it and try to join
> this or a new one?
> Thanks in advance!!
>
> --
> Saludos / Regards.
>
> Analía Lorenzatto.
>
> “It's possible to commit no errors and still lose. That is not weakness.
> That is life". By Captain Jean-Luc Picard.
>
--
Arun
Senior Hadoop/Cassandra Engineer
Cloudwick
Champion of Big Data (Cloudera)
http://www.cloudera.com/content/dev-center/en/home/champions-of-big-data.html
2014 Data Impact Award Winner (Cloudera)
http://www.cloudera.com/content/cloudera/en/campaign/data-impact-awards.html