[jira] [Commented] (CASSANDRA-20250) Provide the ability to disable specific metrics collection

["Benedict Elliott Smith (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/CASSANDRA-20250?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17924059#comment-17924059 ] 

Benedict Elliott Smith commented on CASSANDRA-20250:
----------------------------------------------------

Thanks for filing this ticket [~dnk]. I have for a long time wanted to tackle our extreme inefficiency with metrics, but even I am surprised we're wasting as many as 11% of CPU cycles on metric maintenance on the write path.

I strongly support metrics that can be enabled/disabled (preferably on a live JVM), but we should also consider revisiting our approach to metrics wholesale, as this is really inexcusably inefficient.

> Provide the ability to disable specific metrics collection
> ----------------------------------------------------------
>
>                 Key: CASSANDRA-20250
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-20250
>             Project: Apache Cassandra
>          Issue Type: New Feature
>          Components: Observability/Metrics
>            Reporter: Dmitry Konstantinov
>            Priority: Normal
>         Attachments: cpu_profile_insert.html
>
>
> Cassandra has a lot of metrics collected, many of them are collected per table, so their instance number is multiplied by number of tables. From one side it gives a better observability, from another side metrics are not for free, there is an overhead associated with them:
> 1) CPU overhead: in case of simple CPU bound load: I already see like 5.5% of total CPU spent for metrics in cpu framegraphs for read load and 11% for write load. 
> Example: [^cpu_profile_insert.html] (search by "codahale" pattern)
> 2) memory overhead: we spend memory for entities used to aggregate metrics such as LongAdders and reservoirs + for MBeans (String concatenation within object names is a major cause of it, for each table+metric name combination a new String is created)
>  
> The idea of this ticket is to allow an operator to configure a list of disabled metrics in cassandra.yaml, like:
> {code:java}
> disabled_metrics:
>     - metric_a
>     - metric_b
> {code}
> From implementation point of view I see two possible approaches (which can be combined):
>  # Generic: when a metric is registering if it is listed in disabled_metrics we do not publish it via JMX and provide a noop implementation of metric object (such as histogram) for it.
> Logging analogy: log level check within log method
>  # Specialized: for some metrics the process of value calculation is not for free and introduces an overhead as well, in such cases it would be useful to check within specific logic using an API (like: isMetricEnabled) do we need to do it. Example of such metric: ClientRequestSizeMetrics.recordRowAndColumnCountMetrics
> Logging analogy: an explicit 'if (isDebugEnabled())' condition used when a message parameter is expensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org