You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cassandra.apache.org by Jay Zhuang <ja...@yahoo.com.INVALID> on 2017/07/05 21:32:51 UTC
commitlog_total_space_in_mb tuning
Hi,
commitlog_total_space_in_mb is increased from 1G to 8G in
CASSANDRA-7031. Sometimes we saw the number of dropped mutations spikes.
Not sure if it's a sign that we should increase the
commitlog_total_space_in_mb?
For bean:
org.apache.cassandra.metrics:name=WaitingOnSegmentAllocation,type=CommitLog
Mean is 48684 microseconds
Max is 1386179 microseconds
I think it's relatively high, compare to our other clusters. Does anyone
tune that parameter? Any suggestion on that?
Thanks,
Jay
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.org
For additional commands, e-mail: dev-help@cassandra.apache.org
Re: commitlog_total_space_in_mb tuning
Posted by Nate McCall <na...@thelastpickle.com>.
>
>
> We're running with 128G memory and 30G heap size. Maybe it's good idea
> to increase the commitlog_total_space. On the other hand, even with 8G
> commitlog_total_space, replaying CL after restart takes more than 5
> minutes.
>
> In our case, the actual problem is it's causing lots of read repair
> timeouts as the repair mutations are dropped. Which causes Cassandra JVM
> hang or sometimes crash.
>
Do you have a mix of a small number of really heavily written to tables and
a larger number of tables with fewer writes?
One thing I've had success with when waitingOnSegmentAllocation spiked is
setting memtable_flush_period_in_ms on the less busy tables (obviously not
all the same so you don't flush storm). This seems to keep the
block-and-tackle CL rotation cleaner with fewer tables to flush.
Re: commitlog_total_space_in_mb tuning
Posted by Jay Zhuang <ja...@yahoo.com.INVALID>.
Thanks Jeff for the quick response.
We're running with 3.0.13 which doesn't have commitlog_segment_recycling
option. So it should be disabled.
I think the CL flush is because commitlog full. The commitlog size is
closing to 8G:
#mbean =
org.apache.cassandra.metrics:name=TotalCommitLogSize,type=CommitLog:
Value = 8489271296;
We're running with 128G memory and 30G heap size. Maybe it's good idea
to increase the commitlog_total_space. On the other hand, even with 8G
commitlog_total_space, replaying CL after restart takes more than 5 minutes.
In our case, the actual problem is it's causing lots of read repair
timeouts as the repair mutations are dropped. Which causes Cassandra JVM
hang or sometimes crash.
/Jay
On 7/5/17 2:45 PM, Jeff Jirsa wrote:
>
>
> On 2017-07-05 14:32 (-0700), Jay Zhuang <ja...@yahoo.com.INVALID> wrote:
>> Hi,
>>
>> commitlog_total_space_in_mb is increased from 1G to 8G in
>> CASSANDRA-7031. Sometimes we saw the number of dropped mutations spikes.
>> Not sure if it's a sign that we should increase the
>> commitlog_total_space_in_mb?
>
> 8G seems pretty typical. The real litmus test is in Benedict's comment:
>
>> We can find that we force memtable flushes as a result of log utilisation instead of memtable occupancy quite often
>
> Are your memtable flushes because the memtable is full, or because the commitlog is full?
>
> Also, what version are you running, and do you have segment recycling enabled? (hopefully not: https://issues.apache.org/jira/browse/CASSANDRA-8771 )
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
> For additional commands, e-mail: user-help@cassandra.apache.org
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org
Re: commitlog_total_space_in_mb tuning
Posted by Jeff Jirsa <jj...@apache.org>.
On 2017-07-05 14:32 (-0700), Jay Zhuang <ja...@yahoo.com.INVALID> wrote:
> Hi,
>
> commitlog_total_space_in_mb is increased from 1G to 8G in
> CASSANDRA-7031. Sometimes we saw the number of dropped mutations spikes.
> Not sure if it's a sign that we should increase the
> commitlog_total_space_in_mb?
8G seems pretty typical. The real litmus test is in Benedict's comment:
> We can find that we force memtable flushes as a result of log utilisation instead of memtable occupancy quite often
Are your memtable flushes because the memtable is full, or because the commitlog is full?
Also, what version are you running, and do you have segment recycling enabled? (hopefully not: https://issues.apache.org/jira/browse/CASSANDRA-8771 )
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@cassandra.apache.org
For additional commands, e-mail: user-help@cassandra.apache.org