[jira] [Commented] (KYLIN-1641) Spark - pagination

["wangxianbin (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KYLIN-1641?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15270614#comment-15270614 ] 

wangxianbin commented on KYLIN-1641:
------------------------------------

hey guys, any explain for this?

> Spark - pagination
> ------------------
>
>                 Key: KYLIN-1641
>                 URL: https://issues.apache.org/jira/browse/KYLIN-1641
>             Project: Kylin
>          Issue Type: Improvement
>            Reporter: Dileep
>
> Issue: we have inserted around 10 million records in hive and show the results in web interface through spark dataframe. We cannot get all those 10 million and do the pagination in the front end. So we did the pagination in the spark dataframe using following approach 
>   df1 =df.limit(rowsperPage * pagenumer)
>         df2 = df1.limit(rowsperPage * (pagenumer  -1))
> df1.subtract(df2)).collect().
> This working fine but when we go up the pagenumber (last page ) it is slowing down and not get the results back to front end. 
> Just want to check what we are doing right or any other solution for this problem
> Thanks



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)