[jira] [Commented] (NIFI-2157) Add GenerateTableFetch processor

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15426657#comment-15426657 ] 

ASF GitHub Bot commented on NIFI-2157:
--------------------------------------

Github user simonellistonball commented on the issue:

    https://github.com/apache/nifi/pull/645
  
    The description of partition size here, and indeed the name seems a little unclear. Wouldn't partitionCount make more sense, since that is what it actually represents? As stands, it is not the size, or the number of rows, it is the devisor in the function that gives us rough number of rows.


> Add GenerateTableFetch processor
> --------------------------------
>
>                 Key: NIFI-2157
>                 URL: https://issues.apache.org/jira/browse/NIFI-2157
>             Project: Apache NiFi
>          Issue Type: Sub-task
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>             Fix For: 1.0.0, 1.0.0-Beta
>
>
> This processor would presumably operate like QueryDatabaseTable, except it will contain a "Partition Size" property, and rather than executing the SQL statement(s) to fetch rows, it would generate flow files containing SQL statements that will select rows from a table. If the partition size is indicated, then the SELECT statements will refer to a range of rows, such that each statement will grab only a portion of the table. If max-value columns are specified, then only rows whose observed values for those columns exceed the current maximum will be fetched (i.e. like QueryDatabaseTable). These flow files (due to NIFI-1973) can be passed to ExecuteSQL processors for the actual fetching of rows, and ExecuteSQL can be distributed across cluster nodes and/or multiple tasks. These features enable distributed incremental fetching of rows from database table(s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)