[jira] [Commented] (NIFI-2157) Add GenerateTableFetch processor

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-2157?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15376149#comment-15376149 ] 

ASF GitHub Bot commented on NIFI-2157:
--------------------------------------

GitHub user mattyb149 opened a pull request:

    https://github.com/apache/nifi/pull/645

    NIFI-2157: Add GenerateTableFetch processor

    Refactored common code out of QueryDatabaseTable into an Abstract base class, which involved some refactor of QueryDatabaseTable as well as the added GenerateTableFetch processor.
    
    This includes the addition of the DatabaseAdapter interface and its implementations, going forward this is to isolate database-specific code behind an interface for use by database-related processors.

You can merge this pull request into a Git repository by running:

    $ git pull https://github.com/mattyb149/nifi NIFI-2157_new

Alternatively you can review and apply these changes as the patch at:

    https://github.com/apache/nifi/pull/645.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

    This closes #645
    
----
commit 7b8fd3728f21f6a40b91e401d14738ea9a32fc9e
Author: Matt Burgess <ma...@apache.org>
Date:   2016-07-14T01:29:51Z

    NIFI-2157: Add GenerateTableFetch processor

----


> Add GenerateTableFetch processor
> --------------------------------
>
>                 Key: NIFI-2157
>                 URL: https://issues.apache.org/jira/browse/NIFI-2157
>             Project: Apache NiFi
>          Issue Type: Sub-task
>            Reporter: Matt Burgess
>            Assignee: Matt Burgess
>             Fix For: 1.0.0
>
>
> This processor would presumably operate like QueryDatabaseTable, except it will contain a "Partition Size" property, and rather than executing the SQL statement(s) to fetch rows, it would generate flow files containing SQL statements that will select rows from a table. If the partition size is indicated, then the SELECT statements will refer to a range of rows, such that each statement will grab only a portion of the table. If max-value columns are specified, then only rows whose observed values for those columns exceed the current maximum will be fetched (i.e. like QueryDatabaseTable). These flow files (due to NIFI-1973) can be passed to ExecuteSQL processors for the actual fetching of rows, and ExecuteSQL can be distributed across cluster nodes and/or multiple tasks. These features enable distributed incremental fetching of rows from database table(s).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)