You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Nitan Kainth <ni...@gmail.com> on 2018/06/13 13:41:40 UTC
Timestamp on hints file and system.hints table data
Hi,
I observed a strange behavior about stored hints.
Time stamp of hints file shows several months old. I deleted them and saw
new hints files created with same old date. Why is that?
Also, I see hints files on disk but if I query system.hints table, it shows
0 rows. Why system.hints is not populated?
Version 3.11-1
Re: Timestamp on hints file and system.hints table data
Posted by learner dba <ca...@yahoo.com.INVALID>.
Kurt,
There are no errors in system.log. Just messages about hints sent and played. We will try stop and delete solution and let you know if that fixed the problem. But my curiosity is more about timestamp on hints files, why do we have files with back date; even though I deleted files manually.
On Monday, June 18, 2018, 9:12:16 PM EDT, kurt greaves <ku...@instaclustr.com> wrote:
Send through some examples (and any errors)? Sounds like the file might be corrupt. Not that there's much you can do about that. You can try stopping C*, deleting the file, then starting C* again. You'll have to repair, assuming you haven't repaired already since that hint file was created.
On 18 June 2018 at 13:56, learner dba <ca...@yahoo.com.invalid> wrote:
Yes Kurt, system log is flooded with hints sent and replayed messages.
On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves <ku...@instaclustr.com> wrote:
Not sure what to make of that. Are there any log messages regarding the file and replaying hints? Sounds like maybe it's corrupt (although not sure why it keeps getting rewritten).
On 14 June 2018 at 13:19, Nitan Kainth <ni...@gmail.com> wrote:
Kurt,
Hint file matches UUID matches with another node in the cluster:
-rw-r--r--. 1 root root 6848246 May 13 23:37 1b694180-210a-4b75-8f2a- 748f4a5b6a3d-1526254645089-1. hints
/opt/cassandra/bin/nodetool status |grep 1b694180
UN x.x.x. 23.77 GiB 256 ? 1b694180-210a-4b75-8f2a- 748f4a5b6a3d RAC1
On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <ku...@instaclustr.com> wrote:
Does the UUID on the filename correspond with a UUID in nodetool status?
Sounds to me like it could be something weird with an old node that no longer exists, although hints for old nodes are meant to be cleaned up.
On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
Kurt,
No node is down for months. And yes, I am surprised to look at Unix timestamp on files.
On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months?
On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
Hi,
I observed a strange behavior about stored hints.
Time stamp of hints file shows several months old. I deleted them and saw new hints files created with same old date. Why is that?
Also, I see hints files on disk but if I query system.hints table, it shows 0 rows. Why system.hints is not populated?
Version 3.11-1
Re: Timestamp on hints file and system.hints table data
Posted by kurt greaves <ku...@instaclustr.com>.
Send through some examples (and any errors)? Sounds like the file might be
corrupt. Not that there's much you can do about that. You can try stopping
C*, deleting the file, then starting C* again. You'll have to repair,
assuming you haven't repaired already since that hint file was created.
On 18 June 2018 at 13:56, learner dba <ca...@yahoo.com.invalid>
wrote:
> Yes Kurt, system log is flooded with hints sent and replayed messages.
>
> On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves <
> kurt@instaclustr.com> wrote:
>
>
> Not sure what to make of that. Are there any log messages regarding the
> file and replaying hints? Sounds like maybe it's corrupt (although not sure
> why it keeps getting rewritten).
>
> On 14 June 2018 at 13:19, Nitan Kainth <ni...@gmail.com> wrote:
>
> Kurt,
>
> Hint file matches UUID matches with another node in the cluster:
>
> -rw-r--r--. 1 root root 6848246 May 13 23:37 1b694180-210a-4b75-8f2a-
> 748f4a5b6a3d-1526254645089-1. hints
>
> /opt/cassandra/bin/nodetool status |grep 1b694180
>
> UN x.x.x. 23.77 GiB 256 ? 1b694180-210a-4b75-8f2a-
> 748f4a5b6a3d RAC1
>
>
>
> On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <ku...@instaclustr.com>
> wrote:
>
> Does the UUID on the filename correspond with a UUID in nodetool status?
>
> Sounds to me like it could be something weird with an old node that no
> longer exists, although hints for old nodes are meant to be cleaned up.
>
> On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
>
> Kurt,
>
> No node is down for months. And yes, I am surprised to look at Unix
> timestamp on files.
>
>
>
> On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
>
> system.hints is not used in Cassandra 3. Can't explain the files though,
> are you referring to the files timestamp or the Unix timestamp in the file
> name? Is there a node that's been down for several months?
>
> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
>
> Hi,
>
> I observed a strange behavior about stored hints.
>
> Time stamp of hints file shows several months old. I deleted them and saw
> new hints files created with same old date. Why is that?
>
> Also, I see hints files on disk but if I query system.hints table, it
> shows 0 rows. Why system.hints is not populated?
>
> Version 3.11-1
>
>
>
>
>
Re: Timestamp on hints file and system.hints table data
Posted by learner dba <ca...@yahoo.com.INVALID>.
Yes Kurt, system log is flooded with hints sent and replayed messages.
On Monday, June 18, 2018, 7:30:34 AM EDT, kurt greaves <ku...@instaclustr.com> wrote:
Not sure what to make of that. Are there any log messages regarding the file and replaying hints? Sounds like maybe it's corrupt (although not sure why it keeps getting rewritten).
On 14 June 2018 at 13:19, Nitan Kainth <ni...@gmail.com> wrote:
Kurt,
Hint file matches UUID matches with another node in the cluster:
-rw-r--r--. 1 root root 6848246 May 13 23:37 1b694180-210a-4b75-8f2a- 748f4a5b6a3d-1526254645089-1. hints
/opt/cassandra/bin/nodetool status |grep 1b694180
UN x.x.x. 23.77 GiB 256 ? 1b694180-210a-4b75-8f2a- 748f4a5b6a3d RAC1
On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <ku...@instaclustr.com> wrote:
Does the UUID on the filename correspond with a UUID in nodetool status?
Sounds to me like it could be something weird with an old node that no longer exists, although hints for old nodes are meant to be cleaned up.
On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
Kurt,
No node is down for months. And yes, I am surprised to look at Unix timestamp on files.
On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months?
On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
Hi,
I observed a strange behavior about stored hints.
Time stamp of hints file shows several months old. I deleted them and saw new hints files created with same old date. Why is that?
Also, I see hints files on disk but if I query system.hints table, it shows 0 rows. Why system.hints is not populated?
Version 3.11-1
Re: Timestamp on hints file and system.hints table data
Posted by kurt greaves <ku...@instaclustr.com>.
Not sure what to make of that. Are there any log messages regarding the
file and replaying hints? Sounds like maybe it's corrupt (although not sure
why it keeps getting rewritten).
On 14 June 2018 at 13:19, Nitan Kainth <ni...@gmail.com> wrote:
> Kurt,
>
> Hint file matches UUID matches with another node in the cluster:
>
> -rw-r--r--. 1 root root 6848246 May 13 23:37 1b694180-210a-4b75-8f2a-
> 748f4a5b6a3d-1526254645089-1.hints
>
> /opt/cassandra/bin/nodetool status |grep 1b694180
>
> UN x.x.x. 23.77 GiB 256 ? 1b694180-210a-4b75-8f2a-
> 748f4a5b6a3d RAC1
>
>
>
> On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <ku...@instaclustr.com>
> wrote:
>
>> Does the UUID on the filename correspond with a UUID in nodetool status?
>>
>> Sounds to me like it could be something weird with an old node that no
>> longer exists, although hints for old nodes are meant to be cleaned up.
>>
>> On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
>>
>>> Kurt,
>>>
>>> No node is down for months. And yes, I am surprised to look at Unix
>>> timestamp on files.
>>>
>>>
>>>
>>> On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
>>>
>>> system.hints is not used in Cassandra 3. Can't explain the files though,
>>> are you referring to the files timestamp or the Unix timestamp in the file
>>> name? Is there a node that's been down for several months?
>>>
>>> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>>
>>>> I observed a strange behavior about stored hints.
>>>>
>>>> Time stamp of hints file shows several months old. I deleted them and
>>>> saw new hints files created with same old date. Why is that?
>>>>
>>>> Also, I see hints files on disk but if I query system.hints table, it
>>>> shows 0 rows. Why system.hints is not populated?
>>>>
>>>> Version 3.11-1
>>>>
>>>
>>
>
Re: Timestamp on hints file and system.hints table data
Posted by Nitan Kainth <ni...@gmail.com>.
Kurt,
Hint file matches UUID matches with another node in the cluster:
-rw-r--r--. 1 root root 6848246 May 13 23:37
1b694180-210a-4b75-8f2a-748f4a5b6a3d-1526254645089-1.hints
/opt/cassandra/bin/nodetool status |grep 1b694180
UN x.x.x. 23.77 GiB 256 ?
1b694180-210a-4b75-8f2a-748f4a5b6a3d RAC1
On Thu, Jun 14, 2018 at 12:45 AM, kurt greaves <ku...@instaclustr.com> wrote:
> Does the UUID on the filename correspond with a UUID in nodetool status?
>
> Sounds to me like it could be something weird with an old node that no
> longer exists, although hints for old nodes are meant to be cleaned up.
>
> On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
>
>> Kurt,
>>
>> No node is down for months. And yes, I am surprised to look at Unix
>> timestamp on files.
>>
>>
>>
>> On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
>>
>> system.hints is not used in Cassandra 3. Can't explain the files though,
>> are you referring to the files timestamp or the Unix timestamp in the file
>> name? Is there a node that's been down for several months?
>>
>> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I observed a strange behavior about stored hints.
>>>
>>> Time stamp of hints file shows several months old. I deleted them and
>>> saw new hints files created with same old date. Why is that?
>>>
>>> Also, I see hints files on disk but if I query system.hints table, it
>>> shows 0 rows. Why system.hints is not populated?
>>>
>>> Version 3.11-1
>>>
>>
>
Re: Timestamp on hints file and system.hints table data
Posted by kurt greaves <ku...@instaclustr.com>.
Does the UUID on the filename correspond with a UUID in nodetool status?
Sounds to me like it could be something weird with an old node that no
longer exists, although hints for old nodes are meant to be cleaned up.
On 14 June 2018 at 01:54, Nitan Kainth <ni...@gmail.com> wrote:
> Kurt,
>
> No node is down for months. And yes, I am surprised to look at Unix
> timestamp on files.
>
>
>
> On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
>
> system.hints is not used in Cassandra 3. Can't explain the files though,
> are you referring to the files timestamp or the Unix timestamp in the file
> name? Is there a node that's been down for several months?
>
> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
>
>> Hi,
>>
>> I observed a strange behavior about stored hints.
>>
>> Time stamp of hints file shows several months old. I deleted them and saw
>> new hints files created with same old date. Why is that?
>>
>> Also, I see hints files on disk but if I query system.hints table, it
>> shows 0 rows. Why system.hints is not populated?
>>
>> Version 3.11-1
>>
>
Re: Timestamp on hints file and system.hints table data
Posted by Nitan Kainth <ni...@gmail.com>.
Kurt,
No node is down for months. And yes, I am surprised to look at Unix timestamp on files.
> On Jun 13, 2018, at 6:41 PM, kurt greaves <ku...@instaclustr.com> wrote:
>
> system.hints is not used in Cassandra 3. Can't explain the files though, are you referring to the files timestamp or the Unix timestamp in the file name? Is there a node that's been down for several months?
>
>> On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
>> Hi,
>>
>> I observed a strange behavior about stored hints.
>>
>> Time stamp of hints file shows several months old. I deleted them and saw new hints files created with same old date. Why is that?
>>
>> Also, I see hints files on disk but if I query system.hints table, it shows 0 rows. Why system.hints is not populated?
>>
>> Version 3.11-1
Re: Timestamp on hints file and system.hints table data
Posted by kurt greaves <ku...@instaclustr.com>.
system.hints is not used in Cassandra 3. Can't explain the files though,
are you referring to the files timestamp or the Unix timestamp in the file
name? Is there a node that's been down for several months?
On Wed., 13 Jun. 2018, 23:41 Nitan Kainth, <ni...@gmail.com> wrote:
> Hi,
>
> I observed a strange behavior about stored hints.
>
> Time stamp of hints file shows several months old. I deleted them and saw
> new hints files created with same old date. Why is that?
>
> Also, I see hints files on disk but if I query system.hints table, it
> shows 0 rows. Why system.hints is not populated?
>
> Version 3.11-1
>