[jira] [Commented] (NIFI-10887) Improve Performance of ReplaceText processor

["Mark Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-10887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17640190#comment-17640190 ] 

Mark Payne commented on NIFI-10887:
-----------------------------------

Benchmarks show that we see significant improvements in performance. Screenshots of performance before and after for both Regex Replace and Literal Replace. For both cases, used 4 concurrent tasks and 25 ms Run Duration.

When using Regex Replace, performance was slightly better, as expected.

When using Literal Replace, performance was more pronounced.

> Improve Performance of ReplaceText processor
> --------------------------------------------
>
>                 Key: NIFI-10887
>                 URL: https://issues.apache.org/jira/browse/NIFI-10887
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Mark Payne
>            Assignee: Mark Payne
>            Priority: Major
>              Labels: performance
>             Fix For: 1.20.0
>
>         Attachments: ReplaceText-LiteralReplace-AfterChanges.png, ReplaceText-LiteralReplace-BeforeChanges.png, ReplaceText-RegexReplace-AfterChanges.png, ReplaceText-RegexReplace-BeforeChanges.png
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> When performing some tests with the ReplaceText processor, I found that it seemed to be quite a bit slower than I expected, especially when using a Replacement Strategy of "Literal Replace" and when using a lot of small FlowFiles.
> As a result, I performed some profiling and identified a few areas that could use some improvement:
>  * When using the Literal Replace strategy, we  find matches using {{Pattern.compile(Pattern.quote(...));}} and then using {{{}Pattern.matcher(...).find(){}}}. This is quite inefficient compared to just using {{String.indexOf(...)}} and accounted for approximately 30% of the time spent in the processor.
>  * A significant amount of time was spent flushing the write buffer, as it flushes to disk when finished writing to each individual FlowFile. Even when we set a Run Duration > 0 ms, we flush for each FlowFile. This flush() gets delegated all the way down to the FileOutputStream. However, when using ProcessSession.append(), we intercept this with a NonFlushableOutputStream. We should do this when calling ProcessSession.write() as well. While it makes sense to flush data from the Processor layer's buffer, there's no need to flush past the session layer until the session is committed.
>  * A decent bit of time was spent in the session's get() method calling {{{}final Set<FlowFileRecord> set = unacknowledgedFlowFiles.computeIfAbsent(connection.getFlowFileQueue(), k -> new HashSet<>());{}}}. The time here was spent in StandardFlowFileQueue's hashCode() method, which is the JVM default. We can easily implement hashCode() to just return the hashCode of the identifier, which is a String. This is a pre-computed hashcode so provides constant time of 0 ms (with the exception of the method call itself) so eliminates the expense here.
>  * When using a Run Duration > 0 ms, we can hold InputStreams open by processing multiple FlowFiles in a given Session. This can also significantly improve performance. As such, we should make the default run duration 25 ms instead of 0 ms.
>  * A common pattern with ReplaceText is to prepend text to the beginning of a FlowFile, or line. And then use another ReplaceText to append text to the end of a FlowFile, or line. We should have a strategy for "Surround" that allow us to both Prepend text and Append text. This will result in double the performance for this use case.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)