You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Blake Eggleston (Jira)" <ji...@apache.org> on 2019/12/11 23:30:00 UTC
[jira] [Comment Edited] (CASSANDRA-15388) Add compaction allocation
measurement test to support compaction gc optimization.
[ https://issues.apache.org/jira/browse/CASSANDRA-15388?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16993993#comment-16993993 ]
Blake Eggleston edited comment on CASSANDRA-15388 at 12/11/19 11:29 PM:
------------------------------------------------------------------------
Looking at the code for the GCProfiler in jmh, it looks like the ThreadMXBean is used to measure allocation. ThreadMXBean was one of the first things I looked at, and it wasn’t accurate enough, although it is available as an alternate measurement method in these tests.
AFAIK, ThreadMXBean uses TLAB overflows to approximate allocations with minimum overhead, and the java allocation instrumenter actually invokes the measurement method anytime anything is allocated. So it’s much slower, but super accurate, which is perfect for this use case.
was (Author: bdeggleston):
Looking at the code for GCProfiler, it looks like the ThreadMXBean is used to measure allocation. ThreadMXBean was one of the first things I looked at, and it wasn’t accurate enough, although it is available as an alternate measurement method in these tests.
AFAIK, ThreadMXBean uses TLAB overflows to approximate allocations with minimum overhead, and the java allocation instrumenter actually invokes the measurement method anytime anything is allocated. So it’s much slower, but super accurate, which is perfect for this use case.
> Add compaction allocation measurement test to support compaction gc optimization.
> ----------------------------------------------------------------------------------
>
> Key: CASSANDRA-15388
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15388
> Project: Cassandra
> Issue Type: Sub-task
> Components: Local/Compaction
> Reporter: Blake Eggleston
> Assignee: Blake Eggleston
> Priority: Normal
> Fix For: 4.0
>
>
> This adds a test that is able to quickly and accurately measure the effect of potential gc optimizations against a wide range of (synthetic) compaction workloads. This test accurately measures allocation rates from 16 workloads in less that 2 minutes.
> This test uses google’s {{java-allocation-instrumenter}} agent to measure the workloads. Measurements using this agent are very accurate and pretty repeatable from run to run, with most variance being negligible (1-2 bytes per partition), although workloads with larger but fewer partitions vary a bit more (still less that 0.03%).
> The thinking behind this patch is that with compaction, we’re generally interested in the memory allocated per partition, since garbage scales more or less linearly with the number of partitions compacted. So measuring allocation from a small number of partitions that otherwise represent real world use cases is a good enough approximation.
> In addition to helping with compaction optimizations, this test could be used as a template for future optimization work. This pattern could also be used to set allocation limits on workloads/operations and fail CI if the allocation behavior changes past some threshold.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org