[jira] [Commented] (NIFI-12389) Add variance and standard deviation to AttributeRollingWindow

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-12389?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17787931#comment-17787931 ] 

ASF subversion and git services commented on NIFI-12389:
--------------------------------------------------------

Commit e81d7254ecfa10b893b7686c59d78c2062ed8db1 in nifi's branch refs/heads/main from Matt Burgess
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=e81d7254ec ]

NIFI-12389: Add variance and standard deviation to AttributeRollingWindow

Signed-off-by: Pierre Villard <pi...@gmail.com>

This closes #8048.


> Add variance and standard deviation to AttributeRollingWindow
> -------------------------------------------------------------
>
>                 Key: NIFI-12389
>                 URL: https://issues.apache.org/jira/browse/NIFI-12389
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>            Priority: Major
>             Fix For: 1.latest, 2.latest
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> AttributeRollingWindow currently uses state to calculate metrics like sum, count, and mean (average) for a rolling time window (optionally using sub-windows to use smaller aggregates to estimate metrics for a larger total time window as it stores the window values in state).
> Variance and its positive square root (standard deviation) are other helpful univariate statistics. Using "online" algorithms such as Welford's method adds no extra memory requirement and runs in constant time. Apache Commons Math3 has such an implementation and could be used to add attributes to the FlowFile for variance and standard deviation. These metrics can be used for filtering out noise downstream, for example RouteOnAttribute and/or stateful UpdateAttribute where the current attribute value is more than 2 standard deviations away from the mean.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)