You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@cassandra.apache.org by John Sanda <jo...@gmail.com> on 2017/02/15 01:50:16 UTC

Problems with large partitions and compaction

I have a table that uses LCS and has wound up with partitions upwards of
700 MB. I am seeing lots of the large partition warnings. Client requests
are subsequently failing. The driver is not reporting timeout exception,
just NoHostAvailableExceptions (in the logs I have reviewed so far). I know
that I need to redesign the table to avoid such large partitions. What
specifically goes wrong that results in the instability I am seeing? Or put
another way, what issues will compacting really large partitions cause?
Initially I thought that there was high GC activity, but after closer
inspection that does not really seem to happening. And most of the failures
I am seeing are on reads, but for an entirely different table. Lastly, does
anyone has anyone had success to switching to STCS in this situation as a
work around?

Thanks

- John

Re: Problems with large partitions and compaction

Posted by Dan Kinder <dk...@turnitin.com>.
What Cassandra version? CMS or G1? What are your timeouts set to?

"GC activity"  - Even if there isn't a lot of activity per se maybe there
is a single long pause happening. I have seen large partitions cause lots
of allocation fast.

Looking at SSTable Levels in nodetool cfstats can help, look at it for all
your tables.

Don't recommend switching to STCS until you know more. You end up with
massive compaction that takes a long time to settle down.

On Tue, Feb 14, 2017 at 5:50 PM, John Sanda <jo...@gmail.com> wrote:

> I have a table that uses LCS and has wound up with partitions upwards of
> 700 MB. I am seeing lots of the large partition warnings. Client requests
> are subsequently failing. The driver is not reporting timeout exception,
> just NoHostAvailableExceptions (in the logs I have reviewed so far). I know
> that I need to redesign the table to avoid such large partitions. What
> specifically goes wrong that results in the instability I am seeing? Or put
> another way, what issues will compacting really large partitions cause?
> Initially I thought that there was high GC activity, but after closer
> inspection that does not really seem to happening. And most of the failures
> I am seeing are on reads, but for an entirely different table. Lastly, does
> anyone has anyone had success to switching to STCS in this situation as a
> work around?
>
> Thanks
>
> - John
>



-- 
Dan Kinder
Principal Software Engineer
Turnitin – www.turnitin.com
dkinder@turnitin.com