You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Rene Kochen <Re...@emea.schange.com> on 2011/10/06 11:35:43 UTC
Hinted handoff question
I'am using Cassandra 0.7.7 and have a question about hinted handoff.
I have a cluster of three nodes.
I stop node 3.
I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery.
If I look at the countPendingHints on node 1, it still says 28709.
hinted_handoff_enabled is enabled.
Thanks!
RE: Hinted handoff question
Posted by Rene Kochen <Re...@emea.schange.com>.
I missed that one in the release notes. Thank you!
-----Original Message-----
From: Jonathan Ellis [mailto:jbellis@gmail.com]
Sent: vrijdag 7 oktober 2011 16:39
To: user@cassandra.apache.org
Subject: Re: Hinted handoff question
This was fixed in 0.7.8 by CASSANDRA-2928
In general it's a good idea to upgrade to the latest stable release
before spending a lot of time debugging :)
On Fri, Oct 7, 2011 at 8:33 AM, Rene Kochen
<Re...@emea.schange.com> wrote:
> If I trigger hint delivery using JMX, it works. I see in the log:
>
>
>
> 2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for
> endpoint /172.16.108.19
>
> However, it does not work automatically (i.e. after node 3 is up again).
>
>
>
> I tested it with Cassandra 0.8.6 and there it works as expected.
> Unfortunately, we have 0.7.7. in production.
>
>
>
> Thanks!
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 22:51
>
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> At debug level you should see some log messages such as.
>
>
>
> "Checking remote schema before delivering hints."
>
> "Sleeping {}ms to stagger hint delivery."
>
> "Endpoint {} died before hint delivery, aborting."
>
> "Started hinted handoff for endpoint."
>
>
>
> If you want to move on, you can deliver the hints using JMX or the node
> should start to deliver them if you restart it (i think).
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 11:40 PM, Rene Kochen wrote:
>
> Node 3 is up (using ring on node 1).
>
>
>
> There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
>
>
>
> This is the log from node 1 when nodes 3 starts:
>
>
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted,
> now UP again
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal,
> token 170141183460469000000000000000000000000
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to
> normal
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes
> -> empty pending ranges for Traxis
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to
> /172.16.108.19
>
> 2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
>
> 2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
>
> 2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
>
>
>
> The countPendingHints on node 1 is {172.16.108.19=5176}
>
>
>
> Thanks
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 12:13
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> Check that node 1 sees node 3 as UP (via ring).
>
>
>
> Check the tpstats on node 1, is there an active HH task ?
>
>
>
> Take another crawl through the logs.
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
>
>
> I'am using Cassandra 0.7.7 and have a question about hinted handoff.
>
>
>
> I have a cluster of three nodes.
>
> I stop node 3.
>
> I see that the hint count for node 3 increases on node 1 (countPendingHints
> = 28709).
>
> However, when I start node 3 again, I cannot see anything in the log
> regarding hinted handoff delivery.
>
> If I look at the countPendingHints on node 1, it still says 28709.
>
>
>
> hinted_handoff_enabled is enabled.
>
>
>
> Thanks!
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Hinted handoff question
Posted by Jonathan Ellis <jb...@gmail.com>.
This was fixed in 0.7.8 by CASSANDRA-2928
In general it's a good idea to upgrade to the latest stable release
before spending a lot of time debugging :)
On Fri, Oct 7, 2011 at 8:33 AM, Rene Kochen
<Re...@emea.schange.com> wrote:
> If I trigger hint delivery using JMX, it works. I see in the log:
>
>
>
> 2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for
> endpoint /172.16.108.19
>
> However, it does not work automatically (i.e. after node 3 is up again).
>
>
>
> I tested it with Cassandra 0.8.6 and there it works as expected.
> Unfortunately, we have 0.7.7. in production.
>
>
>
> Thanks!
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 22:51
>
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> At debug level you should see some log messages such as…
>
>
>
> "Checking remote schema before delivering hints…"
>
> "Sleeping {}ms to stagger hint delivery…"
>
> "Endpoint {} died before hint delivery, aborting…"
>
> "Started hinted handoff for endpoint…"
>
>
>
> If you want to move on, you can deliver the hints using JMX or the node
> should start to deliver them if you restart it (i think).
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 11:40 PM, Rene Kochen wrote:
>
> Node 3 is up (using ring on node 1).
>
>
>
> There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
>
>
>
> This is the log from node 1 when nodes 3 starts:
>
>
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted,
> now UP again
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal,
> token 170141183460469000000000000000000000000
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to
> normal
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes
> -> empty pending ranges for Traxis
>
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to
> /172.16.108.19
>
> 2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
>
> 2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
>
> 2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
>
>
>
> The countPendingHints on node 1 is {172.16.108.19=5176}
>
>
>
> Thanks
>
>
>
> Rene
>
>
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 12:13
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
>
>
> Check that node 1 sees node 3 as UP (via ring).
>
>
>
> Check the tpstats on node 1, is there an active HH task ?
>
>
>
> Take another crawl through the logs.
>
>
>
> Cheers
>
>
>
> -----------------
>
> Aaron Morton
>
> Freelance Cassandra Developer
>
> @aaronmorton
>
> http://www.thelastpickle.com
>
>
>
> On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
>
>
> I'am using Cassandra 0.7.7 and have a question about hinted handoff.
>
>
>
> I have a cluster of three nodes.
>
> I stop node 3.
>
> I see that the hint count for node 3 increases on node 1 (countPendingHints
> = 28709).
>
> However, when I start node 3 again, I cannot see anything in the log
> regarding hinted handoff delivery.
>
> If I look at the countPendingHints on node 1, it still says 28709.
>
>
>
> hinted_handoff_enabled is enabled.
>
>
>
> Thanks!
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
RE: Hinted handoff question
Posted by Rene Kochen <Re...@emea.schange.com>.
If I trigger hint delivery using JMX, it works. I see in the log:
2011-10-07 15:17:51,216 INFO 15:17:51,216 Started hinted handoff for endpoint /172.16.108.19
However, it does not work automatically (i.e. after node 3 is up again).
I tested it with Cassandra 0.8.6 and there it works as expected. Unfortunately, we have 0.7.7. in production.
Thanks!
Rene
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: donderdag 6 oktober 2011 22:51
To: user@cassandra.apache.org
Subject: Re: Hinted handoff question
At debug level you should see some log messages such as...
"Checking remote schema before delivering hints..."
"Sleeping {}ms to stagger hint delivery..."
"Endpoint {} died before hint delivery, aborting..."
"Started hinted handoff for endpoint..."
If you want to move on, you can deliver the hints using JMX or the node should start to deliver them if you restart it (i think).
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 11:40 PM, Rene Kochen wrote:
Node 3 is up (using ring on node 1).
There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
This is the log from node 1 when nodes 3 starts:
2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted, now UP again
2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal, token 170141183460469000000000000000000000000
2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to normal
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes -> empty pending ranges for Traxis
2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to /172.16.108.19
2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
The countPendingHints on node 1 is {172.16.108.19=5176}
Thanks
Rene
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: donderdag 6 oktober 2011 12:13
To: user@cassandra.apache.org<ma...@cassandra.apache.org>
Subject: Re: Hinted handoff question
Check that node 1 sees node 3 as UP (via ring).
Check the tpstats on node 1, is there an active HH task ?
Take another crawl through the logs.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
I'am using Cassandra 0.7.7 and have a question about hinted handoff.
I have a cluster of three nodes.
I stop node 3.
I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery.
If I look at the countPendingHints on node 1, it still says 28709.
hinted_handoff_enabled is enabled.
Thanks!
Re: Hinted handoff question
Posted by aaron morton <aa...@thelastpickle.com>.
At debug level you should see some log messages such as…
"Checking remote schema before delivering hints…"
"Sleeping {}ms to stagger hint delivery…"
"Endpoint {} died before hint delivery, aborting…"
"Started hinted handoff for endpoint…"
If you want to move on, you can deliver the hints using JMX or the node should start to deliver them if you restart it (i think).
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 11:40 PM, Rene Kochen wrote:
> Node 3 is up (using ring on node 1).
>
> There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
>
> This is the log from node 1 when nodes 3 starts:
>
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted, now UP again
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal, token 170141183460469000000000000000000000000
> 2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to normal
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes -> empty pending ranges for Traxis
> 2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to /172.16.108.19
> 2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
> 2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
> 2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
>
> The countPendingHints on node 1 is {172.16.108.19=5176}
>
> Thanks
>
> Rene
>
> From: aaron morton [mailto:aaron@thelastpickle.com]
> Sent: donderdag 6 oktober 2011 12:13
> To: user@cassandra.apache.org
> Subject: Re: Hinted handoff question
>
> Check that node 1 sees node 3 as UP (via ring).
>
> Check the tpstats on node 1, is there an active HH task ?
>
> Take another crawl through the logs.
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Cassandra Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
>
>
> I'am using Cassandra 0.7.7 and have a question about hinted handoff.
>
> I have a cluster of three nodes.
> I stop node 3.
> I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
> However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery.
> If I look at the countPendingHints on node 1, it still says 28709.
>
> hinted_handoff_enabled is enabled.
>
> Thanks!
RE: Hinted handoff question
Posted by Rene Kochen <Re...@emea.schange.com>.
Node 3 is up (using ring on node 1).
There is no HH task (active = 0, pending = 0, completed = 0, blocked = 0).
This is the log from node 1 when nodes 3 starts:
2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 has restarted, now UP again
2011-10-06 12:36:54,985DEBUG 12:36:54,985 Node /172.16.108.19 state normal, token 170141183460469000000000000000000000000
2011-10-06 12:36:54,985 INFO 12:36:54,985 Node /172.16.108.19 state jump to normal
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 clearing cached endpoints
2011-10-06 12:36:54,985DEBUG 12:36:54,985 No bootstrapping or leaving nodes -> empty pending ranges for Traxis
2011-10-06 12:36:54,985DEBUG 12:36:54,985 attempting to connect to /172.16.108.19
2011-10-06 12:37:06,672DEBUG 12:37:06,672 Disseminating load info ...
2011-10-06 12:38:06,688DEBUG 12:38:06,688 Disseminating load info ...
2011-10-06 12:39:06,703DEBUG 12:39:06,703 Disseminating load info ...
The countPendingHints on node 1 is {172.16.108.19=5176}
Thanks
Rene
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: donderdag 6 oktober 2011 12:13
To: user@cassandra.apache.org
Subject: Re: Hinted handoff question
Check that node 1 sees node 3 as UP (via ring).
Check the tpstats on node 1, is there an active HH task ?
Take another crawl through the logs.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
I'am using Cassandra 0.7.7 and have a question about hinted handoff.
I have a cluster of three nodes.
I stop node 3.
I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery.
If I look at the countPendingHints on node 1, it still says 28709.
hinted_handoff_enabled is enabled.
Thanks!
Re: Hinted handoff question
Posted by aaron morton <aa...@thelastpickle.com>.
Check that node 1 sees node 3 as UP (via ring).
Check the tpstats on node 1, is there an active HH task ?
Take another crawl through the logs.
Cheers
-----------------
Aaron Morton
Freelance Cassandra Developer
@aaronmorton
http://www.thelastpickle.com
On 6/10/2011, at 10:35 PM, Rene Kochen wrote:
> I'am using Cassandra 0.7.7 and have a question about hinted handoff.
>
> I have a cluster of three nodes.
> I stop node 3.
> I see that the hint count for node 3 increases on node 1 (countPendingHints = 28709).
> However, when I start node 3 again, I cannot see anything in the log regarding hinted handoff delivery.
> If I look at the countPendingHints on node 1, it still says 28709.
>
> hinted_handoff_enabled is enabled.
>
> Thanks!