You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@drill.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2022/09/28 02:06:00 UTC

[jira] [Commented] (DRILL-8320) Prevent Infinite Pagination for Index Paginator

    [ https://issues.apache.org/jira/browse/DRILL-8320?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17610324#comment-17610324 ] 

ASF GitHub Bot commented on DRILL-8320:
---------------------------------------

cgivre opened a new pull request, #2660:
URL: https://github.com/apache/drill/pull/2660

   # [DRILL-8320](https://issues.apache.org/jira/browse/DRILL-8320): Prevent Infinite Pagination for Index Paginator
   
   ## Description
   In some cases that use keyset/index pagination, if the API does not have a boolean column that indicates when to stop, Drill will send requests until the API stops returning data.  This PR fixes this by making the boolean parameter optional.  
   
   If that parameter is not present, if the index result is blank or the same as the previous request, pagination will end.
   
   Note, if the pagination parameters are buried in nested objects, this cannot be configured with a `dataPath`.  If the user uses a `dataPath`, pagination will stop at the first page.
   
   ## Documentation
   No user facing changes.
   
   ## Testing
   Added unit test and tested manually with Hubspot API.




> Prevent Infinite Pagination for Index Paginator
> -----------------------------------------------
>
>                 Key: DRILL-8320
>                 URL: https://issues.apache.org/jira/browse/DRILL-8320
>             Project: Apache Drill
>          Issue Type: Bug
>          Components: Storage - HTTP
>    Affects Versions: 1.20.2
>            Reporter: Charles Givre
>            Assignee: Charles Givre
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> In some cases that use keyset/index pagination, if the API does not have a boolean column that indicates when to stop, Drill will send requests until the API stops returning data.  This PR fixes this by making the boolean parameter optional.  
> If that parameter is not present, if the index result is blank or the same as the previous request, pagination will end.
> Note, if the pagination parameters are buried in nested objects, this cannot be configured with a dataPath.  If the user uses a dataPath, pagination will stop at the first page.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)