You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF subversion and git services (Jira)" <ji...@apache.org> on 2025/03/25 16:45:00 UTC
[jira] [Commented] (NIFI-14313) JoinEnrichment processor; failure to close result sets contributes to excess memory usage
[ https://issues.apache.org/jira/browse/NIFI-14313?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17938281#comment-17938281 ]
ASF subversion and git services commented on NIFI-14313:
--------------------------------------------------------
Commit fc533d05c05fedb927a4b5f0d997f65ad20bf3d5 in nifi's branch refs/heads/main from Jim Steinebrey
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=fc533d05c0 ]
NIFI-14313 JoinEnrichment processor; failure to close result sets contributes to excess memory usage
Signed-off-by: Matt Burgess <ma...@apache.org>
This closes #9821
> JoinEnrichment processor; failure to close result sets contributes to excess memory usage
> -----------------------------------------------------------------------------------------
>
> Key: NIFI-14313
> URL: https://issues.apache.org/jira/browse/NIFI-14313
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 2.0.0, 2.1.0, 2.2.0
> Reporter: Jim Steinebrey
> Assignee: Jim Steinebrey
> Priority: Major
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This ticket involves the JoinEnrichment processor when configured to use a Join Strategy of SQL (JOIN_SQL in the code). When the JoinEnrichment processor uses a SqlJoinStrategy, up to 25 result sets are retained in memory in a cache. In the case of large ResultSets (for example, from record-oriented FlowFiles with many records), this has the potential to adversely impact memory available to the NiFi process.
> When JoinEnrichment uses the SqlJoinStrategy, the SqlJoinStrategy holds a cache of SqlJoinCalciteParameters. Each SqlJoinCalciteParameters holds a PreparedStatement which holds in memory the last result set returned by executing the PreparedStatement.
> The remediation is to close the result set AFTER it is finished being used and BEFORE it is put into the cache.
> This reduces memory usage without worsening performance because a new result set is always created using executeQuery() each time the PreparedStatement is reused.
> The cache is being used to reuse the PreparedStatement to avoid having to compile the sql statement over and over for each flow file processed.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)