[jira] [Updated] (NIFI-8002) Create new Elasticsearch Scroll/Paginated Query processor using REST API

["Joe Gresock (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/NIFI-8002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Joe Gresock updated NIFI-8002:
------------------------------
    Fix Version/s: 1.15.0
       Resolution: Fixed
           Status: Resolved  (was: Patch Available)

> Create new Elasticsearch Scroll/Paginated Query processor using REST API
> ------------------------------------------------------------------------
>
>                 Key: NIFI-8002
>                 URL: https://issues.apache.org/jira/browse/NIFI-8002
>             Project: Apache NiFi
>          Issue Type: Improvement
>    Affects Versions: 1.12.1
>            Reporter: Chris Sampson
>            Priority: Minor
>             Fix For: 1.15.0
>
>         Attachments: ES_Paginated_Queries.json, ES_Paginated_Queries.xml
>
>   Original Estimate: 5h
>          Time Spent: 57.5h
>  Remaining Estimate: 0h
>
> The existing ScrollElasticsearch processor does not fully work with newer versions of Elasticsearch (e.g. 7.x).
> A new version of this processor should be created using the REST API approach, allowing for a query to be passed as input FlowFile content. Appropriate parameters should be set as processor properties (e.g. index, scroll size, scroll time) and allow for additional query string parameters to be set as dynamic properties (see NIFI-8001).
> The processor should support both [_scroll|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#scroll-search-results] and [_pit/search_after|https://www.elastic.co/guide/en/elasticsearch/reference/current/paginate-search-results.html#search-after]. While scroll may be backward compatible,  _pit was introduced in ES 7 (an earlier version of search_before exists for <7, but suggest not trying to support that as part of this ticket).
> It may be possible to simply extend the existing REST API's Query processor to page through results with the selected method, rather than create a new processor.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)