[jira] [Commented] (NIFI-3431) Support batch update in Notify processor

["Jul Tomten (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376449#comment-17376449 ] 

Jul Tomten commented on NIFI-3431:
----------------------------------

I'm on NiFi 1.13.2

I have a WAIT processor and 2000 files queud up in upstream connection

The problem is that the WAIT processor uses more than 100% CPU all the time as long as the messages are on queue.
Any idea about the problem?

Release Signal Identifier = ${uuid}

Target Signal Count = 1

Signal Counter Name = No value set

Wait Buffer Count = 1500

Releasable FlowFile Count = 1

Expiration Duration = 10 min

Attribute Copy Mode = Keep original

Wait Mode = Keep in the upstream connection

Wait Penalty Duration = 5 min

 

I'd planned to have more than 10 000 000 messages on queue and then let them expire after 4 days. Some or all messages may be notified to proceed. Is the WAIT intended to handel such a scenario?

 

> Support batch update in Notify processor
> ----------------------------------------
>
>                 Key: NIFI-3431
>                 URL: https://issues.apache.org/jira/browse/NIFI-3431
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>    Affects Versions: 1.2.0
>            Reporter: Koji Kawamura
>            Assignee: Koji Kawamura
>            Priority: Major
>             Fix For: 1.2.0
>
>
> NIFI-3216 added ability to wait for N signals. It supports waiting for N fragments split by SplitXXXX processors. However, since Notify processor has to increase count one by one by calling expensive replace cache operation over network, it doesn't provide a practical performance when user configured a flow looks like below as N glow:
> {code}
> Split
>  --> (original) -> Wait for N
>  --> (split) -> Do something -> Notify 1
> {code}
> This JIRA improves Notify processor by:
> - Add "Signal Buffer Count" to specify max number of flow files that can be buffered and update cache at once
> - Add "Signal Counter Delta" to specify delta grater than 1, EL supported
> "Signal Buffer Count" would be useful in a flow like this:
> {code}
> Split
>  --> (original) -> Wait for N
>  --> (split) -> Filter or something -> Notify M
> {code}
> Buffer incoming M flow files and perform cache replace operation once.
> So does "Signal Counter Delta":
> {code}
> Split
>  --> (original) -> Wait for N
>  --> (split) -> Filter or something -> Merge M -> PutXXX -> Notify M
> {code}
> Specify 'M' via Attribute Expression Language.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)