You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Stephen Pierce <sp...@verifyle.com> on 2012/10/18 20:10:49 UTC
Hinted Handoff runs every ten minutes
I installed Cassandra on three nodes. I then ran a test suite against them to generate load. The test suite is designed to generate the same type of load that we plan to have in production. As one of many tests, I reset one of the nodes to check the failure/recovery modes. Cassandra worked just fine.
I stopped the load generation, and got distracted with some other project/problem. A few days later, I noticed something strange on one of the nodes. On this node hinted handoff starts every ten minutes, and while it seems to finish without any errors, it will be started again in ten minutes. None of the nodes has any traffic, and hasn't for several days. I checked the logs, and this goes back to the initial failure/recovery testing:
INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java (line 294) Started hinted handoff for token: 113427455640312821154458202477256070484 with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java (line 294) Started hinted handoff for token: 113427455640312821154458202477256070484 with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java (line 294) Started hinted handoff for token: 113427455640312821154458202477256070484 with IP: /192.168.128.136
INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
The other nodes are happy and don't show this behavior. All the test data is readable, and everything is fine, but I'm curious why hinted handoff is running on one node all the time.
I searched the bug database, and I found a bug that seems to have the same symptoms:
https://issues.apache.org/jira/browse/CASSANDRA-3733
Although it's been marked fixed in 0.6, this describes my problem exactly.
I'm running Cassandra 1.1.5 from Datastax on Centos 6.0:
http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
Is anyone else seeing this behavior? What can I do to provide more information?
Steve
Re: Hinted Handoff runs every ten minutes
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Hi!
I am having the same issue on 1.0.8.
Checked number of SSTables, on two nodes I have 1 (on each) and on 1 node I
have none.
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Mon, Oct 22, 2012 at 1:44 AM, aaron morton <aa...@thelastpickle.com>wrote:
> I *think* this may be ghost rows which have not being compacted.
>
> How many SSTables are on disk for the HintedHandoff CF ?
>
> Cheers
>
> -----------------
> Aaron Morton
> Freelance Developer
> @aaronmorton
> http://www.thelastpickle.com
>
> On 19/10/2012, at 7:16 AM, David Daeschler <da...@gmail.com>
> wrote:
>
> Hi Steve,
>
> Also confirming this. After having a node go down on Cassandra 1.0.8
> there seems to be hinted handoff between two of our 4 nodes every 10
> minutes. Our setup also shows 0 rows. It does not appear to have any
> effect on the operation of the ring, just fills up the log files.
>
> - David
>
>
>
> On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <sp...@verifyle.com>
> wrote:
>
> I installed Cassandra on three nodes. I then ran a test suite against them
> to generate load. The test suite is designed to generate the same type of
> load that we plan to have in production. As one of many tests, I reset one
> of the nodes to check the failure/recovery modes. Cassandra worked just
> fine.
>
>
>
> I stopped the load generation, and got distracted with some other
> project/problem. A few days later, I noticed something strange on one of
> the
> nodes. On this node hinted handoff starts every ten minutes, and while it
> seems to finish without any errors, it will be started again in ten
> minutes.
> None of the nodes has any traffic, and hasn’t for several days. I checked
> the logs, and this goes back to the initial failure/recovery testing:
>
>
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
>
>
> The other nodes are happy and don’t show this behavior. All the test data
> is
> readable, and everything is fine, but I’m curious why hinted handoff is
> running on one node all the time.
>
>
>
> I searched the bug database, and I found a bug that seems to have the same
> symptoms:
>
> https://issues.apache.org/jira/browse/CASSANDRA-3733
>
> Although it’s been marked fixed in 0.6, this describes my problem exactly.
>
>
>
> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
>
>
> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
>
>
>
> Is anyone else seeing this behavior? What can I do to provide more
> information?
>
>
>
> Steve
>
>
>
>
>
> --
> David Daeschler
>
>
>
Re: Hinted Handoff runs every ten minutes
Posted by aaron morton <aa...@thelastpickle.com>.
Thanks.
I thought it had been addressed so before but couldn't find the ticket.
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 25/10/2012, at 12:56 AM, Brandon Williams <dr...@gmail.com> wrote:
> On Sun, Oct 21, 2012 at 6:44 PM, aaron morton <aa...@thelastpickle.com> wrote:
>> I *think* this may be ghost rows which have not being compacted.
>
> You would be correct in the case of 1.0.8:
> https://issues.apache.org/jira/browse/CASSANDRA-3955
>
> -Brandon
Re: Hinted Handoff runs every ten minutes
Posted by Roshan <co...@gmail.com>.
You can delete the hints from JConsole by using HintedHadOffManager MBean.
/Roshan
--
View this message in context: http://cassandra-user-incubator-apache-org.3065146.n2.nabble.com/Hinted-Handoff-runs-every-ten-minutes-tp7583263p7583378.html
Sent from the cassandra-user@incubator.apache.org mailing list archive at Nabble.com.
Re: Hinted Handoff runs every ten minutes
Posted by Tamar Fraenkel <ta...@tok-media.com>.
Is there a walk around other than upgrade?
Thanks,
*Tamar Fraenkel *
Senior Software Engineer, TOK Media
[image: Inline image 1]
tamar@tok-media.com
Tel: +972 2 6409736
Mob: +972 54 8356490
Fax: +972 2 5612956
On Wed, Oct 24, 2012 at 1:56 PM, Brandon Williams <dr...@gmail.com> wrote:
> On Sun, Oct 21, 2012 at 6:44 PM, aaron morton <aa...@thelastpickle.com>
> wrote:
> > I *think* this may be ghost rows which have not being compacted.
>
> You would be correct in the case of 1.0.8:
> https://issues.apache.org/jira/browse/CASSANDRA-3955
>
> -Brandon
>
Re: Hinted Handoff runs every ten minutes
Posted by Vegard Berget <po...@fantasista.no>.
Hi,
HintsColumnFamily directory on Node 1 (the first to be upgraded):1.8K
Oct 27 11:27 system-HintsColumnFamily-hf-2-Data.db79 Oct 27 11:27
system-HintsColumnFamily-hf-2-Digest.sha1496 Oct 27 11:27
system-HintsColumnFamily-hf-2-Filter.db26 Oct 27 11:27
system-HintsColumnFamily-hf-2-Index.db4.3K Oct 27 11:27
system-HintsColumnFamily-hf-2-Statistics.db
HintsColumnFamily-directory on Node 2 (the second to be upgraded):12K
Oct 27 11:37 system-HintsColumnFamily-hf-3-Data.db79 Oct 27 11:37
system-HintsColumnFamily-hf-3-Digest.sha1976 Oct 27 11:37
system-HintsColumnFamily-hf-3-Filter.db11 Oct 27 11:37
system-HintsColumnFamily-hf-3-Index.db4.3K Oct 27 11:37
system-HintsColumnFamily-hf-3-Statistics.db
11:37 is exactly ten minutes after the node was started after
upgrading, according to the logs, which means that the first node's
sstables is generated when the second node starts.
The sstable2json is more or less equal for both nodes, but node 1
starts with "40000000000000000000000000000000" and node 2 with
"00".The rest looks like this:"00": {"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXX":
{"deletedAt": -9223372036854775808, "subColumns": []},
"f7XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX": {"deletedAt":
-9223372036854775808, "subColumns": []},(The f7XXXXXXXXXXXXXXXXX is
repeated with multiple different keys, but all are above
f7XXXXXXXXXXXXXXXXXXXXX. The deletedAt is always equal - on both
nodes)
Of course, this could be caused by me doing a mistake while upgrading.
I have not tried to compact, but repair runs every night. It's a
low traffic cluster to test cassandra, if that matters.
.vegard,
----- Original Message -----
From: user@cassandra.apache.org
To:
Cc:
Sent:Sun, 11 Nov 2012 13:16:01 -0600
Subject:Re: Hinted Handoff runs every ten minutes
How many hint sstables are there? What does sstable2json show?
On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner wrote:
> Is there a ticket open for this for 1.1.6?
>
> We also noticed this after upgrading from 1.1.3 to 1.1.6. Every
node runs a
> 0 row hinted handoff every 10 minutes. N-1 nodes hint to the same
node,
> while that node hints to another node.
>
>
> On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget wrote:
>>
>> Hi,
>>
>> I have the exact same problem with 1.16. HintsColumnFamily
consists of
>> one row (Rowkey 00, nothing more). The "problem" started after
upgrading
>> from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts
and
>> finishes after sending "0 rows".
>>
>> .vegard,
>>
>>
>>
>> ----- Original Message -----
>> From:
>> user@cassandra.apache.org
>>
>> To:
>>
>> Cc:
>>
>> Sent:
>> Mon, 29 Oct 2012 23:45:30 +0100
>>
>> Subject:
>> Re: Hinted Handoff runs every ten minutes
>>
>>
>> Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
>> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>> >
>> > How can I check to see why it keeps running HintedHandoff?
>> you have tombstone is system.HintsColumnFamily use list command in
>> cassandra-cli to check
>>
>
>
>
> --
>
> Mike Heffner
> Librato, Inc.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Hinted Handoff runs every ten minutes
Posted by Jonathan Ellis <jb...@gmail.com>.
How many hint sstables are there? What does sstable2json show?
On Thu, Nov 8, 2012 at 3:23 PM, Mike Heffner <mi...@librato.com> wrote:
> Is there a ticket open for this for 1.1.6?
>
> We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
> 0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
> while that node hints to another node.
>
>
> On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget <po...@fantasista.no> wrote:
>>
>> Hi,
>>
>> I have the exact same problem with 1.1.6. HintsColumnFamily consists of
>> one row (Rowkey 00, nothing more). The "problem" started after upgrading
>> from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts and
>> finishes after sending "0 rows".
>>
>> .vegard,
>>
>>
>>
>> ----- Original Message -----
>> From:
>> user@cassandra.apache.org
>>
>> To:
>> <us...@cassandra.apache.org>
>> Cc:
>>
>> Sent:
>> Mon, 29 Oct 2012 23:45:30 +0100
>>
>> Subject:
>> Re: Hinted Handoff runs every ten minutes
>>
>>
>> Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
>> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>> >
>> > How can I check to see why it keeps running HintedHandoff?
>> you have tombstone is system.HintsColumnFamily use list command in
>> cassandra-cli to check
>>
>
>
>
> --
>
> Mike Heffner <mi...@librato.com>
> Librato, Inc.
>
>
--
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com
Re: Hinted Handoff runs every ten minutes
Posted by Mike Heffner <mi...@librato.com>.
Is there a ticket open for this for 1.1.6?
We also noticed this after upgrading from 1.1.3 to 1.1.6. Every node runs a
0 row hinted handoff every 10 minutes. N-1 nodes hint to the same node,
while that node hints to another node.
On Tue, Oct 30, 2012 at 1:35 PM, Vegard Berget <po...@fantasista.no> wrote:
> Hi,
>
> I have the exact same problem with 1.1.6. HintsColumnFamily consists of
> one row (Rowkey 00, nothing more). The "problem" started after upgrading
> from 1.1.4 to 1.1.6. Every ten minutes HintedHandoffManager starts and
> finishes after sending "0 rows".
>
> .vegard,
>
>
>
> ----- Original Message -----
> From:
> user@cassandra.apache.org
>
> To:
> <us...@cassandra.apache.org>
> Cc:
>
> Sent:
> Mon, 29 Oct 2012 23:45:30 +0100
>
> Subject:
> Re: Hinted Handoff runs every ten minutes
>
>
> Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
> >
> > How can I check to see why it keeps running HintedHandoff?
> you have tombstone is system.HintsColumnFamily use list command in
> cassandra-cli to check
>
>
--
Mike Heffner <mi...@librato.com>
Librato, Inc.
Re[2]: Hinted Handoff runs every ten minutes
Posted by Marcus Both <mb...@terra.com.br>.
When I did the upgrade from 1.0.9 to 1.1.6, I had this same issue.
And then I fixed it with the following steps below, in each of the nodes.
[default@unknown] use system;
Authenticated to keyspace: system
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: 55555555555555555555555555555554
1 Row Returned.
Elapsed time: 84 msec(s).
[default@system] del HintsColumnFamily[55555555555555555555555555555554];
system keyspace is not user-modifiable.
InvalidRequestException(why:system keyspace is not user-modifiable.)
at org.apache.cassandra.thrift.Cassandra$remove_result.read(Cassandra.java:18213)
at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
at org.apache.cassandra.thrift.Cassandra$Client.recv_remove(Cassandra.java:861)
at org.apache.cassandra.thrift.Cassandra$Client.remove(Cassandra.java:845)
at org.apache.cassandra.cli.CliClient.executeDelete(CliClient.java:455)
at org.apache.cassandra.cli.CliClient.executeCLIStatement(CliClient.java:225)
at org.apache.cassandra.cli.CliMain.processStatementInteractive(CliMain.java:219)
at org.apache.cassandra.cli.CliMain.main(CliMain.java:346)
[default@system]
[root@ ~]# /usr/local/apache-cassandra/bin/nodetool -h 127.0.0.1 scrub system HintsColumnFamily
[root@ ~]# /usr/local/apache-cassandra/bin/cassandra-cli -h 127.0.0.1 -p 9160
Connected to: "nosql-world" on 127.0.0.1/9160
Welcome to Cassandra CLI version 1.1.6
Type 'help;' or '?' for help.
Type 'quit;' or 'exit;' to quit.
[default@unknown] use system;
Authenticated to keyspace: system
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
0 Row Returned.
Elapsed time: 73 msec(s).
[default@system]
tail /var/log/cassandra/system.log
INFO [CompactionExecutor:193] 2012-11-06 12:03:12,233 OutputHandler.java (line 42) Scrubbing SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-11-Data.db')
INFO [CompactionExecutor:193] 2012-11-06 12:03:12,236 OutputHandler.java (line 42) Scrub of SSTableReader(path='/var/lib/cassandra/data/system/HintsColumnFamily/system-HintsColumnFamily-hf-11-Data.db') complete; looks like all 1 rows were tombstoned
--
Marcus Both
On Mon, 5 Nov 2012 19:21:40 +0000
Stephen Pierce <sp...@verifyle.com> wrote:
> I did have a tombstone. I couldn't seem to delete it, so I truncated the column family.
>
> Thanks for the help.
>
> Steve
>
> [default@system] list HintsColumnFamily;
> Using default limit of 100
> Using default column limit of 100
> -------------------
> RowKey: 55555555555555555555555555555554
>
> I wasn't able to delete the row:
>
> [default@system] del HintsColumnFamily[55555555555555555555555555555554];
> row removed.
> [default@system] list HintsColumnFamily;
> Using default limit of 100
> Using default column limit of 100
> -------------------
> RowKey: 55555555555555555555555555555554
>
> But I finally truncated the column family:
>
> [default@system] truncate HintsColumnFamily;
> HintsColumnFamily truncated.
>
>
> -----Original Message-----
> From: Radim Kolar [mailto:hsn@filez.com]
> Sent: Monday, October 29, 2012 3:46 PM
> To: user@cassandra.apache.org
> Subject: Re: Hinted Handoff runs every ten minutes
>
> Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
> > I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
> >
> > How can I check to see why it keeps running HintedHandoff?
> you have tombstone is system.HintsColumnFamily use list command in cassandra-cli to check
>
--
Marcus Both
RE: Hinted Handoff runs every ten minutes
Posted by Stephen Pierce <sp...@verifyle.com>.
I did have a tombstone. I couldn't seem to delete it, so I truncated the column family.
Thanks for the help.
Steve
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: 55555555555555555555555555555554
I wasn't able to delete the row:
[default@system] del HintsColumnFamily[55555555555555555555555555555554];
row removed.
[default@system] list HintsColumnFamily;
Using default limit of 100
Using default column limit of 100
-------------------
RowKey: 55555555555555555555555555555554
But I finally truncated the column family:
[default@system] truncate HintsColumnFamily;
HintsColumnFamily truncated.
-----Original Message-----
From: Radim Kolar [mailto:hsn@filez.com]
Sent: Monday, October 29, 2012 3:46 PM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
> I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>
> How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in cassandra-cli to check
Re: Hinted Handoff runs every ten minutes
Posted by Vegard Berget <po...@fantasista.no>.
Hi,
I have the exact same problem with 1.1.6. HintsColumnFamily consists
of one row (Rowkey 00, nothing more). The "problem" started after
upgrading from 1.1.4 to 1.1.6. Every ten minutes
HintedHandoffManager starts and finishes after sending "0 rows".
.vegard,
----- Original Message -----
From: user@cassandra.apache.org
To:
Cc:
Sent:Mon, 29 Oct 2012 23:45:30 +0100
Subject:Re: Hinted Handoff runs every ten minutes
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
> I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>
> How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in
cassandra-cli to check
Re: Hinted Handoff runs every ten minutes
Posted by Radim Kolar <hs...@filez.com>.
Dne 29.10.2012 23:24, Stephen Pierce napsal(a):
> I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
>
> How can I check to see why it keeps running HintedHandoff?
you have tombstone is system.HintsColumnFamily use list command in
cassandra-cli to check
RE: Hinted Handoff runs every ten minutes
Posted by Stephen Pierce <sp...@verifyle.com>.
I'm running 1.1.5; the bug says it's fixed in 1.0.9/1.1.0.
How can I check to see why it keeps running HintedHandoff?
Steve
-----Original Message-----
From: Brandon Williams [mailto:driftx@gmail.com]
Sent: Wednesday, October 24, 2012 4:56 AM
To: user@cassandra.apache.org
Subject: Re: Hinted Handoff runs every ten minutes
On Sun, Oct 21, 2012 at 6:44 PM, aaron morton <aa...@thelastpickle.com> wrote:
> I *think* this may be ghost rows which have not being compacted.
You would be correct in the case of 1.0.8:
https://issues.apache.org/jira/browse/CASSANDRA-3955
-Brandon
Re: Hinted Handoff runs every ten minutes
Posted by Brandon Williams <dr...@gmail.com>.
On Sun, Oct 21, 2012 at 6:44 PM, aaron morton <aa...@thelastpickle.com> wrote:
> I *think* this may be ghost rows which have not being compacted.
You would be correct in the case of 1.0.8:
https://issues.apache.org/jira/browse/CASSANDRA-3955
-Brandon
Re: Hinted Handoff runs every ten minutes
Posted by aaron morton <aa...@thelastpickle.com>.
I *think* this may be ghost rows which have not being compacted.
How many SSTables are on disk for the HintedHandoff CF ?
Cheers
-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com
On 19/10/2012, at 7:16 AM, David Daeschler <da...@gmail.com> wrote:
> Hi Steve,
>
> Also confirming this. After having a node go down on Cassandra 1.0.8
> there seems to be hinted handoff between two of our 4 nodes every 10
> minutes. Our setup also shows 0 rows. It does not appear to have any
> effect on the operation of the ring, just fills up the log files.
>
> - David
>
>
>
> On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <sp...@verifyle.com> wrote:
>> I installed Cassandra on three nodes. I then ran a test suite against them
>> to generate load. The test suite is designed to generate the same type of
>> load that we plan to have in production. As one of many tests, I reset one
>> of the nodes to check the failure/recovery modes. Cassandra worked just
>> fine.
>>
>>
>>
>> I stopped the load generation, and got distracted with some other
>> project/problem. A few days later, I noticed something strange on one of the
>> nodes. On this node hinted handoff starts every ten minutes, and while it
>> seems to finish without any errors, it will be started again in ten minutes.
>> None of the nodes has any traffic, and hasn’t for several days. I checked
>> the logs, and this goes back to the initial failure/recovery testing:
>>
>>
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
>> (line 294) Started hinted handoff for token:
>> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>>
>> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
>> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>>
>>
>>
>> The other nodes are happy and don’t show this behavior. All the test data is
>> readable, and everything is fine, but I’m curious why hinted handoff is
>> running on one node all the time.
>>
>>
>>
>> I searched the bug database, and I found a bug that seems to have the same
>> symptoms:
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-3733
>>
>> Although it’s been marked fixed in 0.6, this describes my problem exactly.
>>
>>
>>
>> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
>>
>> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
>>
>>
>>
>> Is anyone else seeing this behavior? What can I do to provide more
>> information?
>>
>>
>>
>> Steve
>>
>>
>
>
>
> --
> David Daeschler
Re: Hinted Handoff runs every ten minutes
Posted by David Daeschler <da...@gmail.com>.
Hi Steve,
Also confirming this. After having a node go down on Cassandra 1.0.8
there seems to be hinted handoff between two of our 4 nodes every 10
minutes. Our setup also shows 0 rows. It does not appear to have any
effect on the operation of the ring, just fills up the log files.
- David
On Thu, Oct 18, 2012 at 2:10 PM, Stephen Pierce <sp...@verifyle.com> wrote:
> I installed Cassandra on three nodes. I then ran a test suite against them
> to generate load. The test suite is designed to generate the same type of
> load that we plan to have in production. As one of many tests, I reset one
> of the nodes to check the failure/recovery modes. Cassandra worked just
> fine.
>
>
>
> I stopped the load generation, and got distracted with some other
> project/problem. A few days later, I noticed something strange on one of the
> nodes. On this node hinted handoff starts every ten minutes, and while it
> seems to finish without any errors, it will be started again in ten minutes.
> None of the nodes has any traffic, and hasn’t for several days. I checked
> the logs, and this goes back to the initial failure/recovery testing:
>
>
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,618 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:19:26,779 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,622 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:29:26,735 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,624 HintedHandOffManager.java
> (line 294) Started hinted handoff for token:
> 113427455640312821154458202477256070484 with IP: /192.168.128.136
>
> INFO [HintedHandoff:1] 2012-10-18 10:39:26,751 HintedHandOffManager.java
> (line 390) Finished hinted handoff of 0 rows to endpoint /192.168.128.136
>
>
>
> The other nodes are happy and don’t show this behavior. All the test data is
> readable, and everything is fine, but I’m curious why hinted handoff is
> running on one node all the time.
>
>
>
> I searched the bug database, and I found a bug that seems to have the same
> symptoms:
>
> https://issues.apache.org/jira/browse/CASSANDRA-3733
>
> Although it’s been marked fixed in 0.6, this describes my problem exactly.
>
>
>
> I’m running Cassandra 1.1.5 from Datastax on Centos 6.0:
>
> http://rpm.datastax.com/community/noarch/apache-cassandra11-1.1.5-1.noarch.rpm
>
>
>
> Is anyone else seeing this behavior? What can I do to provide more
> information?
>
>
>
> Steve
>
>
--
David Daeschler