You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Ezra Stuetzel <ez...@riskiq.net> on 2016/09/16 22:12:25 UTC
large system hint partition
What would be the likely causes of large system hint partitions? Normally
large partition warnings are for user defined tables which they are writing
large partitions to. In this case, it appears C* is writing large
partitions to the system.hints table. Gossip is not backed up.
version: C* 2.2.7
WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
BigTableWriter.java:184 - Writing large partition
system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
Thanks,
Ezra
Re: large system hint partition
Posted by Romain Hardouin <ro...@yahoo.fr>.
Hi,
> More recent (I think 2.2) don't have this problem since they write hints to the file system as per the commit log
Flat files hints were implemented starting from 3.0 https://issues.apache.org/jira/browse/CASSANDRA-6230
Best,
Romain
Re: large system hint partition
Posted by Graham Sanderson <gr...@vast.com>.
The reason for large partitions is that the partition key is just the uuid of the target node
More recent (I think 2.2) don't have this problem since they write hints to the file system as per the commit log
Sadly the large partitions make things worse when you are hinting hence presumably under stress
Sent from my iPhone
> On Sep 16, 2016, at 6:13 PM, Nicolas Douillet <ni...@gmail.com> wrote:
>
> Hi Erza,
>
> Have you a dead node in your cluster?
> Because the coordinator stores a hint about dead replicas in the local system.hints when a node is dead or didn't respond to a write request.
>
> --
> Nicolas
>
>
>
>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ez...@riskiq.net> a écrit :
>> What would be the likely causes of large system hint partitions? Normally large partition warnings are for user defined tables which they are writing large partitions to. In this case, it appears C* is writing large partitions to the system.hints table. Gossip is not backed up.
>>
>> version: C* 2.2.7
>> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220 BigTableWriter.java:184 - Writing large partition system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>
>>
>>
>> Thanks,
>>
>> Ezra
Re: large system hint partition
Posted by Ezra Stuetzel <ez...@riskiq.net>.
Yeah I tried that, but oddly the table had nothing in it.
I changed the compaction strategy from leveled to sizetierd and ran a major
compaction on each node. I haven't seen the message logged on any node in a
few days which makes me think that fixed it because it is normally logged
multiple times per day.
On Sun, Sep 18, 2016 at 4:29 AM, Carlos Alonso <in...@mrcalonso.com> wrote:
> By inspecting the contents on your system.hints table, specifically the
> host_id column, you can see which is the destination host of those hints
> and check if it is one of the alive or dead ones.
>
> Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
>
> On 18 September 2016 at 04:35, Ezra Stuetzel <ez...@riskiq.net>
> wrote:
>
>> Hey Nicolas,
>>
>> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
>> both show 4 healthy nodes. In the past we have had some nodes we eliminated
>> by using 'nodetool assassinate'. However, I checked system.peers table on
>> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
>> appear that any nodes have any awareness of an unreachable node which could
>> be causing hints to back up. Any ideas for further troubleshooting what the
>> hints are?
>>
>> Thanks,
>> Ezra
>>
>> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
>> nicolas.douillet@gmail.com> wrote:
>>
>>> Hi Erza,
>>>
>>> Have you a dead node in your cluster?
>>> Because the coordinator stores a hint about dead replicas in the local
>>> system.hints when a node is dead or didn't respond to a write request.
>>>
>>> --
>>> Nicolas
>>>
>>>
>>>
>>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ez...@riskiq.net>
>>> a écrit :
>>>
>>>> What would be the likely causes of large system hint partitions?
>>>> Normally large partition warnings are for user defined tables which they
>>>> are writing large partitions to. In this case, it appears C* is writing
>>>> large partitions to the system.hints table. Gossip is not backed up.
>>>>
>>>> version: C* 2.2.7
>>>>
>>>> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>>>> BigTableWriter.java:184 - Writing large partition
>>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>>>
>>>>
>>>> Thanks,
>>>>
>>>> Ezra
>>>>
>>>
>>
>
Re: large system hint partition
Posted by Carlos Alonso <in...@mrcalonso.com>.
By inspecting the contents on your system.hints table, specifically the
host_id column, you can see which is the destination host of those hints
and check if it is one of the alive or dead ones.
Carlos Alonso | Software Engineer | @calonso <https://twitter.com/calonso>
On 18 September 2016 at 04:35, Ezra Stuetzel <ez...@riskiq.net>
wrote:
> Hey Nicolas,
>
> There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
> both show 4 healthy nodes. In the past we have had some nodes we eliminated
> by using 'nodetool assassinate'. However, I checked system.peers table on
> all 4 of our nodes and they each show 3 peers as expected. So it doesn't
> appear that any nodes have any awareness of an unreachable node which could
> be causing hints to back up. Any ideas for further troubleshooting what the
> hints are?
>
> Thanks,
> Ezra
>
> On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
> nicolas.douillet@gmail.com> wrote:
>
>> Hi Erza,
>>
>> Have you a dead node in your cluster?
>> Because the coordinator stores a hint about dead replicas in the local
>> system.hints when a node is dead or didn't respond to a write request.
>>
>> --
>> Nicolas
>>
>>
>>
>> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ez...@riskiq.net>
>> a écrit :
>>
>>> What would be the likely causes of large system hint partitions?
>>> Normally large partition warnings are for user defined tables which they
>>> are writing large partitions to. In this case, it appears C* is writing
>>> large partitions to the system.hints table. Gossip is not backed up.
>>>
>>> version: C* 2.2.7
>>>
>>> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>>> BigTableWriter.java:184 - Writing large partition
>>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>>
>>>
>>> Thanks,
>>>
>>> Ezra
>>>
>>
>
Re: large system hint partition
Posted by Ezra Stuetzel <ez...@riskiq.net>.
Hey Nicolas,
There are no dead nodes. 'nodetool status' and 'nodetool describecluster'
both show 4 healthy nodes. In the past we have had some nodes we eliminated
by using 'nodetool assassinate'. However, I checked system.peers table on
all 4 of our nodes and they each show 3 peers as expected. So it doesn't
appear that any nodes have any awareness of an unreachable node which could
be causing hints to back up. Any ideas for further troubleshooting what the
hints are?
Thanks,
Ezra
On Fri, Sep 16, 2016 at 4:13 PM, Nicolas Douillet <
nicolas.douillet@gmail.com> wrote:
> Hi Erza,
>
> Have you a dead node in your cluster?
> Because the coordinator stores a hint about dead replicas in the local
> system.hints when a node is dead or didn't respond to a write request.
>
> --
> Nicolas
>
>
>
> Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ez...@riskiq.net> a
> écrit :
>
>> What would be the likely causes of large system hint partitions? Normally
>> large partition warnings are for user defined tables which they are writing
>> large partitions to. In this case, it appears C* is writing large
>> partitions to the system.hints table. Gossip is not backed up.
>>
>> version: C* 2.2.7
>>
>> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
>> BigTableWriter.java:184 - Writing large partition
>> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>>
>>
>> Thanks,
>>
>> Ezra
>>
>
Re: large system hint partition
Posted by Nicolas Douillet <ni...@gmail.com>.
Hi Erza,
Have you a dead node in your cluster?
Because the coordinator stores a hint about dead replicas in the local
system.hints when a node is dead or didn't respond to a write request.
--
Nicolas
Le sam. 17 sept. 2016 à 00:12, Ezra Stuetzel <ez...@riskiq.net> a
écrit :
> What would be the likely causes of large system hint partitions? Normally
> large partition warnings are for user defined tables which they are writing
> large partitions to. In this case, it appears C* is writing large
> partitions to the system.hints table. Gossip is not backed up.
>
> version: C* 2.2.7
>
> WARN [MemtableFlushWriter:134] 2016-09-16 04:27:39,220
> BigTableWriter.java:184 - Writing large partition
> system/hints:7ce838aa-f30f-494a-8caa-d44d1440e48b (128181097 bytes)
>
>
> Thanks,
>
> Ezra
>