[jira] [Commented] (NIFI-8605) ExecuteSQLRecord processor consumes a large heap volume when use with PostgreSQL JDBC driver

["ASF subversion and git services (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-8605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17467045#comment-17467045 ] 

ASF subversion and git services commented on NIFI-8605:
-------------------------------------------------------

Commit 0ac8f1b32c5c0194e8e1bc7384bec81ed7500dfc in nifi's branch refs/heads/main from Vibhath Ileperuma
[ https://gitbox.apache.org/repos/asf?p=nifi.git;h=0ac8f1b ]

NIFI-8605 Adding a new property for ExecuteSQL and ExecuteSQLRecord processors to enable/disable auto committing
change the default value of auto commit function to true
Changed the auto commit property name and add more details in the description
If the auto commit is set to false, commit() is called for consistency
adds unit tests
Fix the check style issue of having more than 200 characters in single line

Signed-off-by: Matthew Burgess <ma...@apache.org>

This closes #5554


> ExecuteSQLRecord processor consumes a large heap volume when use with PostgreSQL JDBC driver
> --------------------------------------------------------------------------------------------
>
>                 Key: NIFI-8605
>                 URL: https://issues.apache.org/jira/browse/NIFI-8605
>             Project: Apache NiFi
>          Issue Type: Bug
>            Reporter: Vibhath Arunapriya Ileperuma
>            Assignee: Vibhath Arunapriya Ileperuma
>            Priority: Major
>              Labels: Beginner, beginner
>         Attachments: GC.LOG
>
>          Time Spent: 1h 50m
>  Remaining Estimate: 0h
>
> I'm using ExecuteSQLRecord processor to query from PostgreSQL. A 'select' query I'm using can return more than 60 million rows. I have configured the fetch size to 1000 to avoid fetching all the data into memory at once.
> But when the processor is started, heap starts to grow very fast. I have configured to NIFI to have 50GB heap size and even that amount is filled within minutes. When the heap is filled Garbage collector tries to clean the heap blocking other threads. 
> It seems like NIFI loads all the data to memory even though fetch size is set to 1000. I have attached the NIFI's GC log here with this ticket for reference.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)