You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2021/02/23 16:01:00 UTC
[jira] [Commented] (NIFI-8222) When processing a lot of small
FlowFiles, Provenance Repo spends most of its time in lock contention. That
can be improved.
[ https://issues.apache.org/jira/browse/NIFI-8222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17289154#comment-17289154 ]
ASF subversion and git services commented on NIFI-8222:
-------------------------------------------------------
Commit 528fce2407d092d4ced1a58fcc14d0bc6e660b89 in nifi's branch refs/heads/main from Mark Payne
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=528fce2 ]
NIFI-7646, NIFI-8222: Instead of having StandardProcessSession call ContentRepository.read(ContentClaim), introduced a new ContentRepository.read(ResourceClaim) and hold open the InputStream to the ResourceClaim. This can't be supported by EncryptedContentRepository, so introduced a method to allow using this or not. The benefit here is that when we have many FlowFiles read within a session, such as when using MergeContent/MergeRecord or a processor configured with a Run Duration, we can hold open a single InputStream instead of constantly opening FileInputStreams and seeking to the appropriate location. This is much faster.
- Instead of entering a 'synchronized' block for every provenance event, serialize up to 1 MB worth of data, and then enter synchronized block to write that data out. This avoids large amounts of lock contention and context switches
NIFI-7646: Removed TODO and unused Jackson dependency
Signed-off-by: Matthew Burgess <ma...@apache.org>
This closes #4818
> When processing a lot of small FlowFiles, Provenance Repo spends most of its time in lock contention. That can be improved.
> ---------------------------------------------------------------------------------------------------------------------------
>
> Key: NIFI-8222
> URL: https://issues.apache.org/jira/browse/NIFI-8222
> Project: Apache NiFi
> Issue Type: Improvement
> Components: Core Framework
> Reporter: Mark Payne
> Assignee: Mark Payne
> Priority: Major
> Fix For: 1.14.0, 1.13.1
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)