You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by Oleg Dulin <ol...@gmail.com> on 2013/11/04 20:34:49 UTC
1.1.11: system keyspace is filling up
I have a dual DC setup, 4 nodes, RF=4 in each.
The one that is used as primary has its system keyspace fill up with
200 gigs of data, majority of which is hints.
Why does this happen ?
How can I clean it up ?
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: 1.1.11: system keyspace is filling up
Posted by Aaron Morton <aa...@thelastpickle.com>.
> What happens if they are not being successfully delivered ? Will they eventually TTL-out ?
They have a TTL set to the gc_grace_seconds on the CF at the time of the write.
I’ve also seen hints build up in multi DC systems due to timeouts on the coordinator. i.e. the remote nodes are up, co-ordinator starts the writes, remote nodes process the request (no dropped messages), but the response is lost. These are tracked as timeouts on the MessagingServiceMBean.
Cheers
-----------------
Aaron Morton
New Zealand
@aaronmorton
Co-Founder & Principal Consultant
Apache Cassandra Consulting
http://www.thelastpickle.com
On 22/11/2013, at 6:00 pm, Rahul Menon <ra...@apigee.com> wrote:
> Oleg,
>
> The system keyspace is not replicated it is local to the node. You should check your logs to see if there are Timeouts from streaming hints, i believe the default value to stream hints it 10 seconds. When i ran into this problem i truncated hints to clear out the space and then ran a repair so ensure that all the data was consistant across all nodes, even if there was a failure.
>
> -rm
>
>
> On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <ol...@gmail.com> wrote:
> What happens if they are not being successfully delivered ? Will they eventually TTL-out ?
>
>
>
> Also, do I need to truncate hints on every node or is it replicated ?
>
>
>
> Oleg
>
>
>
> On 2013-11-04 21:34:55 +0000, Robert Coli said:
>
>
>
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <ol...@gmail.com> wrote:
>
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
>
>
> The one that is used as primary has its system keyspace fill up with 200 gigs of data, majority of which is hints.
>
>
>
> Why does this happen ?
>
>
>
> How can I clean it up ?
>
>
>
> If you have this many hints, you probably have flapping / frequent network partition, or very overloaded nodes. If you compare the number of hints to the number of dropped messages, that would be informative. If you're hinting because you're dropping, increase capacity. If you're hinting because of partition, figure out why there's so much partition.
>
>
>
> WRT cleaning up hints, they will automatically be cleaned up eventually, as long as they are successfully being delivered. If you need to manually clean them up you can truncate system.hints keyspace.
>
>
>
> =Rob
>
>
>
>
>
>
>
>
> --
>
> Regards,
>
> Oleg Dulin
>
> http://www.olegdulin.com
>
>
Re: 1.1.11: system keyspace is filling up
Posted by Rahul Menon <ra...@apigee.com>.
Oleg,
The system keyspace is not replicated it is local to the node. You should
check your logs to see if there are Timeouts from streaming hints, i
believe the default value to stream hints it 10 seconds. When i ran into
this problem i truncated hints to clear out the space and then ran a repair
so ensure that all the data was consistant across all nodes, even if there
was a failure.
-rm
On Tue, Nov 5, 2013 at 6:29 PM, Oleg Dulin <ol...@gmail.com> wrote:
> What happens if they are not being successfully delivered ? Will they
> eventually TTL-out ?
>
>
> Also, do I need to truncate hints on every node or is it replicated ?
>
>
> Oleg
>
>
> On 2013-11-04 21:34:55 +0000, Robert Coli said:
>
>
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <ol...@gmail.com> wrote:
>
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
>
> The one that is used as primary has its system keyspace fill up with 200
> gigs of data, majority of which is hints.
>
>
> Why does this happen ?
>
>
> How can I clean it up ?
>
>
> If you have this many hints, you probably have flapping / frequent network
> partition, or very overloaded nodes. If you compare the number of hints to
> the number of dropped messages, that would be informative. If you're
> hinting because you're dropping, increase capacity. If you're hinting
> because of partition, figure out why there's so much partition.
>
>
> WRT cleaning up hints, they will automatically be cleaned up eventually,
> as long as they are successfully being delivered. If you need to manually
> clean them up you can truncate system.hints keyspace.
>
>
> =Rob
>
>
>
>
>
>
> --
>
> Regards,
>
> Oleg Dulin
>
> http://www.olegdulin.com
>
Re: 1.1.11: system keyspace is filling up
Posted by Oleg Dulin <ol...@gmail.com>.
What happens if they are not being successfully delivered ? Will they
eventually TTL-out ?
Also, do I need to truncate hints on every node or is it replicated ?
Oleg
On 2013-11-04 21:34:55 +0000, Robert Coli said:
> On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <ol...@gmail.com> wrote:
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
> The one that is used as primary has its system keyspace fill up with
> 200 gigs of data, majority of which is hints.
>
> Why does this happen ?
>
> How can I clean it up ?
>
> If you have this many hints, you probably have flapping / frequent
> network partition, or very overloaded nodes. If you compare the number
> of hints to the number of dropped messages, that would be informative.
> If you're hinting because you're dropping, increase capacity. If you're
> hinting because of partition, figure out why there's so much partition.
>
> WRT cleaning up hints, they will automatically be cleaned up
> eventually, as long as they are successfully being delivered. If you
> need to manually clean them up you can truncate system.hints keyspace.
>
> =Rob
>
--
Regards,
Oleg Dulin
http://www.olegdulin.com
Re: 1.1.11: system keyspace is filling up
Posted by Robert Coli <rc...@eventbrite.com>.
On Mon, Nov 4, 2013 at 11:34 AM, Oleg Dulin <ol...@gmail.com> wrote:
> I have a dual DC setup, 4 nodes, RF=4 in each.
>
> The one that is used as primary has its system keyspace fill up with 200
> gigs of data, majority of which is hints.
>
> Why does this happen ?
>
> How can I clean it up ?
If you have this many hints, you probably have flapping / frequent network
partition, or very overloaded nodes. If you compare the number of hints to
the number of dropped messages, that would be informative. If you're
hinting because you're dropping, increase capacity. If you're hinting
because of partition, figure out why there's so much partition.
WRT cleaning up hints, they will automatically be cleaned up eventually, as
long as they are successfully being delivered. If you need to manually
clean them up you can truncate system.hints keyspace.
=Rob