[jira] [Commented] (NIFI-10167) Allow tracking of detailed metrics such as CPU Usage by processors

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-10167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17561216#comment-17561216 ] 

ASF subversion and git services commented on NIFI-10167:
--------------------------------------------------------

Commit 1f18345929bcf60047010b1584366f3981ca8494 in nifi's branch refs/heads/main from David Handermann
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=1f18345929 ]

NIFI-10163 Corrected StandardProcessSessionTest after NIFI-10167


> Allow tracking of detailed metrics such as CPU Usage by processors
> ------------------------------------------------------------------
>
>                 Key: NIFI-10167
>                 URL: https://issues.apache.org/jira/browse/NIFI-10167
>             Project: Apache NiFi
>          Issue Type: New Feature
>          Components: Core Framework
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>             Fix For: 1.17.0
>
>          Time Spent: 1h 40m
>  Remaining Estimate: 0h
>
> NiFi offers a great deal of detail on the canvas about how much data was read/written in the past 5 minutes, number of tasks run, and how much time that took.
> However, a question we are often asked is "which processors are using up all of my CPU?" From looking at tools like {{top}} we can see that NiFi is using a lot of CPU. And we can guess, based on the fact that Processor A spent 50 minutes of Processing time over the past 5 minutes that Processor A is likely using a lot of resources. But we cannot really know for sure how much CPU it is using vs. disk/socket I/O waits, etc.
> This task will introduce more detailed metrics such as how many seconds of CPU time a processor has consumed, how much time it has spent reading/writing to the content repo, etc. These metrics should be exposed via the {{nifi.sh diagnostics}} command.
> We should make clear both how much of a Processor's time is spent using CPU, as well as what percentage of the total CPU used by NiFi was spent for a given Processor. This will significantly help understand the resource usage characteristics of a given component.
> However, tracking this may be expensive. So we should provide the ability to turn this feature on/off and ideally also allow for sampling of metrics and extrapolating out those numbers so that we can monitor these things only for a percentage of invocations and then give reasonable estimations based on that.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)