[jira] [Commented] (NIFI-5601) Add fragment.* attributes to GenerateTableFetch

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-5601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672242#comment-16672242 ] 

ASF GitHub Bot commented on NIFI-5601:
--------------------------------------

Github user patricker commented on the issue:

    https://github.com/apache/nifi/pull/3074
  
    +1 LG - Reviewed, ran tests, built and ran the changes locally against a larger table. 


> Add fragment.* attributes to GenerateTableFetch
> -----------------------------------------------
>
>                 Key: NIFI-5601
>                 URL: https://issues.apache.org/jira/browse/NIFI-5601
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Extensions
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>            Priority: Major
>
> GenerateTableFetch will generate SQL statements in multiple flow files corresponding to "pages" of data for a single execution of the processor. QueryDatabaseTable does something similar, although it also does the fetching of the rows and adds fragment.* attributes to the outgoing flow files, in case a merge or other correlation needs to be done later.
> GenerateTableFetch (although it doesn't produce source data per se) is usually used in conjunction with ExecuteSQL to fetch the actual pages of rows, and sometimes these would need to be merged/correlated. To that end, this Jira proposes to add the same fragment.* attributes to the outgoing GTF flow files for use downstream.
> Note that if the SQL statements are distributed among a cluster (for parallel execution by ExecuteSQL), they will not be able to be merged, as different nodes may get different fragments and often no single node will get all the fragments.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)