You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@geode.apache.org by "Bill Burcham (Jira)" <ji...@apache.org> on 2021/10/26 22:00:00 UTC
[jira] [Updated] (GEODE-9002) Add New Statistic Type For
/proc/schedstat
[ https://issues.apache.org/jira/browse/GEODE-9002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Burcham updated GEODE-9002:
--------------------------------
Description:
Linux performance icon Brendan Gregg advocates the [USE|http://www.brendangregg.com/usemethod.html] method of performance analysis: Utilization Saturation and Errors.
When it comes to CPU, Geode captures a number of _utilization_ statistics. Some are direct like LinuxSystemStats cpuIdle and cpuActive. Others are indirect like:
* DistributionStats
** heartbeatsSent: you may see a gap in the every-five-seconds heartbeats
* StatSampler
** delayDuration: you may see a rise when CPU is scarce
** sampleCount: you may see an interruption in the regular once-per-second sampling
* (G1GC collector)
** (various memory utilization statistics may indicate memory pressure which in turn can give rise to long GC pauses)
* LinuxSystemStats
** cpuSteal: indicating that the virtualization environment has not given the VM its share of CPU
But utilization statistics alone can't tell you when a resource (like CPU) is _saturated_, i.e. when demand is higher than the servicing ability. If you're just looking at utilization metrics, then a saturated system might look a lot like a system just below saturation. In order to tell the difference, saturation metrics are needed.
In the case of CPU, there is a conceptual queue in front of each processor. Tasks (operating system threads) that are ready to run, enter a queue, and after some delay, are given a time slice by an actual physical CPU.
You might think that Geode's LinuxSystemStats loadAverage1 and 5 and 15, might fit this bill. Those statistics do provide some saturation information. The problem is, they conflate CPU with I/O and other things (see [Linux Load Averages: Solving the Mystery|[http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html].)]
A better, more specific measure of CPU saturation is available through statistics exposed via the /proc/schedstat virtual file.
When this ticket is complete, there will be a new statistic type called LinuxThreadScheduler, with -three- four associated statistics gathered directly from /proc/schedstat or derived from data gathered from it:
* runningTimeNanos: sum of all time spent running by tasks on this processor in nanoseconds
* queuedTimeNanos: sum of all time spent waiting to run by tasks on this processor in nanoseconds
* tasksScheduledCount: # of tasks (not necessarily unique) given to the processor
* meanTaskQueuedTimeNanos: average time that a ready-to-run task waited for a CPU, since the last sample, in nanoseconds
One "statistic" will be gathered for each CPU. So a Geode process running on a two-CPU system will capture two statistics, called "cpu0", "cpu1", each of this new type.
By default Geode will not gather these new statistics. A TBD Java system property will be used to enable gathering the new LinuxThreadScheduler statistic.
was:
Linux performance icon Brendan Gregg advocates the [USE|http://www.brendangregg.com/usemethod.html] method of performance analysis: Utilization Saturation and Errors.
When it comes to CPU, Geode captures a number of _utilization_ statistics. Some are direct like LinuxSystemStats cpuIdle and cpuActive. Others are indirect like:
* DistributionStats
** heartbeatsSent: you may see a gap in the every-five-seconds heartbeats
* StatSampler
** delayDuration: you may see a rise when CPU is scarce
** sampleCount: you may see an interruption in the regular once-per-second sampling
* (G1GC collector)
** (various memory utilization statistics may indicate memory pressure which in turn can give rise to long GC pauses)
* LinuxSystemStats
** cpuSteal: indicating that the virtualization environment has not given the VM its share of CPU
But utilization statistics alone can't tell you when a resource (like CPU) is _saturated_, i.e. when demand is higher than the servicing ability. If you're just looking at utilization metrics, then a saturated system might look a lot like a system just below saturation. In order to tell the difference, saturation metrics are needed.
In the case of CPU, there is a conceptual queue in front of each processor. Tasks (operating system threads) that are ready to run, enter a queue, and after some delay, are given a time slice by an actual physical CPU.
You might think that Geode's LinuxSystemStats loadAverage1 and 5 and 15, might fit this bill. Those statistics do provide some saturation information. The problem is, they conflate CPU with I/O and other things (see [Linux Load Averages: Solving the Mystery|[http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html].)]
A better, more specific measure of CPU saturation is available through statistics exposed via the /proc/schedstat virtual file.
When this ticket is complete, there will be a new statistic type called LinuxThreadScheduler, with three associated statistics gathered directly from /proc/schedstat or derived from data gathered from it:
* runningTimeNanos: sum of all time spent running by tasks on this processor in nanoseconds
* queuedTimeNanos: sum of all time spent waiting to run by tasks on this processor in nanoseconds
* tasksScheduledCount: # of tasks (not necessarily unique) given to the processor
* meanTaskQueuedTimeNanos: average time that a ready-to-run task waited for a CPU, since the last sample, in nanoseconds
One "statistic" will be gathered for each CPU. So a Geode process running on a two-CPU system will capture two statistics, called "cpu0", "cpu1", each of this new type.
By default Geode will not gather these new statistics. A TBD Java system property will be used to enable gathering the new LinuxThreadScheduler statistic.
> Add New Statistic Type For /proc/schedstat
> ------------------------------------------
>
> Key: GEODE-9002
> URL: https://issues.apache.org/jira/browse/GEODE-9002
> Project: Geode
> Issue Type: New Feature
> Components: statistics
> Reporter: Bill Burcham
> Assignee: Bill Burcham
> Priority: Major
> Labels: pull-request-available
>
> Linux performance icon Brendan Gregg advocates the [USE|http://www.brendangregg.com/usemethod.html] method of performance analysis: Utilization Saturation and Errors.
> When it comes to CPU, Geode captures a number of _utilization_ statistics. Some are direct like LinuxSystemStats cpuIdle and cpuActive. Others are indirect like:
> * DistributionStats
> ** heartbeatsSent: you may see a gap in the every-five-seconds heartbeats
> * StatSampler
> ** delayDuration: you may see a rise when CPU is scarce
> ** sampleCount: you may see an interruption in the regular once-per-second sampling
> * (G1GC collector)
> ** (various memory utilization statistics may indicate memory pressure which in turn can give rise to long GC pauses)
> * LinuxSystemStats
> ** cpuSteal: indicating that the virtualization environment has not given the VM its share of CPU
>
> But utilization statistics alone can't tell you when a resource (like CPU) is _saturated_, i.e. when demand is higher than the servicing ability. If you're just looking at utilization metrics, then a saturated system might look a lot like a system just below saturation. In order to tell the difference, saturation metrics are needed.
> In the case of CPU, there is a conceptual queue in front of each processor. Tasks (operating system threads) that are ready to run, enter a queue, and after some delay, are given a time slice by an actual physical CPU.
> You might think that Geode's LinuxSystemStats loadAverage1 and 5 and 15, might fit this bill. Those statistics do provide some saturation information. The problem is, they conflate CPU with I/O and other things (see [Linux Load Averages: Solving the Mystery|[http://www.brendangregg.com/blog/2017-08-08/linux-load-averages.html].)]
> A better, more specific measure of CPU saturation is available through statistics exposed via the /proc/schedstat virtual file.
> When this ticket is complete, there will be a new statistic type called LinuxThreadScheduler, with -three- four associated statistics gathered directly from /proc/schedstat or derived from data gathered from it:
> * runningTimeNanos: sum of all time spent running by tasks on this processor in nanoseconds
> * queuedTimeNanos: sum of all time spent waiting to run by tasks on this processor in nanoseconds
> * tasksScheduledCount: # of tasks (not necessarily unique) given to the processor
> * meanTaskQueuedTimeNanos: average time that a ready-to-run task waited for a CPU, since the last sample, in nanoseconds
> One "statistic" will be gathered for each CPU. So a Geode process running on a two-CPU system will capture two statistics, called "cpu0", "cpu1", each of this new type.
> By default Geode will not gather these new statistics. A TBD Java system property will be used to enable gathering the new LinuxThreadScheduler statistic.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)