[jira] [Commented] (NIFI-15634) Full query execution during metadata discovery for Generic/MySQL in QueryDatabaseTableRecord causes high heap usage.

["Sejal Gupta (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-15634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18124969#comment-18124969 ] 

Sejal Gupta commented on NIFI-15634:
------------------------------------

Hi [~gauravkadaskar] , submitted Pull Request [#11781| https://github.com/apache/nifi/pull/11781] to address this issue.

Summary of Verification:
Tested with a MySQL database using general_log to inspect executed queries during runtime. As shown in the attached screenshot, metadata discovery now correctly executes `SELECT * FROM large_table WHERE 1 = 0` prior to executing the actual data extraction query `SELECT * FROM large_table`. This prevents full table scans and high heap memory usage during schema inspection.

> Full query execution during metadata discovery for Generic/MySQL in QueryDatabaseTableRecord causes high heap usage.
> --------------------------------------------------------------------------------------------------------------------
>
>                 Key: NIFI-15634
>                 URL: https://issues.apache.org/jira/browse/NIFI-15634
>             Project: Apache NiFi
>          Issue Type: Bug
>    Affects Versions: 2.7.2
>            Reporter: gaurav kadaskar
>            Priority: Major
>              Labels: heap, querydatabasetablerecord
>         Attachments: Screenshot from 2026-10-08 15-18-11.png
>
>          Time Spent: 10m
>  Remaining Estimate: 0h
>
> h3. Problem
> When Database Type is set to Generic or MySQL in QueryDatabaseTableRecord,
> getDatabaseDialectService() returns DatabaseAdapterDatabaseDialectService.
> During metadata discovery, NiFi executes the full user query instead of using a
> metadata-only query. Since no guard condition is applied, the query scans the
> entire dataset, which leads to high heap usage and JVM memory pressure for
> large result sets.
> h3. Previous Behavior
> Earlier NiFi versions generated a metadata-safe query by appending:
> {code:sql}
> WHERE 1=0
> {code}
> This allowed schema retrieval without reading actual data.
> h3. Expected Behavior
> Metadata discovery should use a metadata-only query and must not execute the
> full user query.
> h3. Current Behavior
> Full user query is executed to obtain metadata.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)