[jira] [Created] (HBASE-24543) ScheduledChore logging is too chatty, replace with metrics

["Andrew Kyle Purtell (Jira)" <ji...@apache.org>]

Andrew Kyle Purtell created HBASE-24543:
-------------------------------------------

             Summary: ScheduledChore logging is too chatty, replace with metrics
                 Key: HBASE-24543
                 URL: https://issues.apache.org/jira/browse/HBASE-24543
             Project: HBase
          Issue Type: Improvement
          Components: metrics, Operability
            Reporter: Andrew Kyle Purtell


ScheduledChore logs at DEBUG level the execution time of each chore. 

We used to log an average execution time across all chores every five minutes, which by consensus was judged to not be useful. Derived metrics like averages or histograms should be calculated per chore. So we modified the logging to dump the chore execution time each time it runs, to facilitate such calculations with the log aggregation and searching tool of choice. Per chore execution logging is more useful, in that sense, but may be too chatty. This is not unexpected but let me provide my observations so we can revisit this.

On the master, for example, this is logged every second:
{noformat}
2020-06-11 16:35:28,263 DEBUG [master/apurtell-ltm:8100.splitLogManager..Chore.1] hbase.ScheduledChore: SplitLogManager Timeout Monitor execution time: 0 ms.
{noformat}

Does the value of these lines outweigh the cost of 86,400 log lines per day per master instance? (At least.)

On the regionserver it is somewhat better, these are logged every 10 seconds:
{noformat}
2020-06-11 16:37:57,203 DEBUG [regionserver/apurtell-ltm:8120.Chore.1] hbase.ScheduledChore: CompactionChecker execution time: 0 ms.
2020-06-11 16:37:57,203 DEBUG [regionserver/apurtell-ltm:8120.Chore.1] hbase.ScheduledChore: MemstoreFlusherChore execution time: 0 ms.
{noformat}

So that will be 17,280 log lines per day per regionserver. (At least.)

Perhaps these should be moved to TRACE level. 

We should definitely replace this logging with histogram metrics. There should be a separate metric for each distinct chore classname, allocated as needed. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)