[jira] [Commented] (NIFI-1706) Extend QueryDatabaseTable to support arbitrary queries

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/NIFI-1706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16430286#comment-16430286 ] 

ASF GitHub Bot commented on NIFI-1706:
--------------------------------------

Github user ijokarumawak commented on the issue:

    https://github.com/apache/nifi/pull/2162
  
    @patricker Thanks for the update, and sorry for my delayed response. I just got the chance to review this today.
    
    This has a conflict with #2504 . I can fix compilation level conflict by removing AbstractDatabaseProcessor.onPropertyModified which has been deleted by #2504.
    However, It seems that we need to update other places, too. I encountered few unit test failures at 
    TestGenerateTableFetch:
    ```
    [ERROR]   TestGenerateTableFetch.testAddedRows:202 expected:<4> but was:<0>
    [ERROR]   TestGenerateTableFetch.testAddedRowsRightBounded:402 expected:<4> but was:<0>
    [ERROR]   TestGenerateTableFetch.testAddedRowsTwoTables:276 expected:<1> but was:<0>
    [ERROR]   TestGenerateTableFetch.testAddedRowsWithCustomWhereClause:1112 expected:<5> but was:<0>
    [ERROR]   TestGenerateTableFetch.testMultipleColumnTypeMissing:1250 expected:<1> but was:<0>
    ```
    I'll continue looking at these test failures. It'd be appreciated if you can rebase the PR with the latest master and see how we can fix those, too. Thanks!


> Extend QueryDatabaseTable to support arbitrary queries
> ------------------------------------------------------
>
>                 Key: NIFI-1706
>                 URL: https://issues.apache.org/jira/browse/NIFI-1706
>             Project: Apache NiFi
>          Issue Type: Improvement
>          Components: Core Framework
>    Affects Versions: 1.4.0
>            Reporter: Paul Bormans
>            Assignee: Peter Wicks
>            Priority: Major
>              Labels: features
>
> The QueryDatabaseTable is able to observe a configured database table for new rows and yield these into the flowfile. The model of an rdbms however is often (if not always) normalized so you would need to join various tables in order to "flatten" the data into useful events for a processing pipeline as can be build with nifi or various tools within the hadoop ecosystem.
> The request is to extend the processor to specify an arbitrary sql query instead of specifying the table name + columns.
> In addition (this may be another issue?) it is desired to limit the number of rows returned per run. Not just because of bandwidth issue's from the nifi pipeline onwards but mainly because huge databases may not be able to return so many records within a reasonable time.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)