[GitHub] [spark] HyukjinKwon opened a new pull request #26809: [SPARK-30185][SQL] Implement Dataset.tail API

[GitBox <gi...@apache.org>]

HyukjinKwon opened a new pull request #26809: [SPARK-30185][SQL] Implement Dataset.tail API
URL: https://github.com/apache/spark/pull/26809
 
 
   ### What changes were proposed in this pull request?
   
   This PR proposes a `tail` API. 
   
   Namely, as below:
   
   ```scala
   scala> spark.range(10).head(5)
   res1: Array[Long] = Array(0, 1, 2, 3, 4)
   scala> spark.range(10).tail(5)
   res2: Array[Long] = Array(5, 6, 7, 8, 9)
   ```
   
   Implementation details will be similar with `head` but it will be reversed:
   
   1. Run the job against the last partition and collect rows. If this is enough, return as is.
   2. If this is not enough, calculate the number of partitions to select more based upon
    `spark.sql.limit.scaleUpFactor`
   3. Run more jobs against more partitions (in a reversed order compared to head) as many as the number calculated from 2.
   4. Go to 2.
   
   
   ### Why are the changes needed?
   
   Many other systems support the way to take data from the end, for instance, pandas[1] and
    Python[2][3]. Scala collections APIs also have head and tail
   
   On the other hand, in Spark, we only provide a way to take data from the start
    (e.g., DataFrame.head).
   
   This has been requested multiple times here and there in Spark user mailing list[4], StackOverFlow[5][6], JIRA[7] and other third party projects such as
    Koalas[8]. In addition, this missing API seems explicitly mentioned in comparison to another system[9] time to time.
   
   It seems we're missing non-trivial use case in Spark and this motivated me to propose this API.
   
   [1] https://pandas.pydata.org/pandas-docs/stable/reference/api/pandas.DataFrame.tail.html?highlight=tail#pandas.DataFrame.tail
   [2] https://stackoverflow.com/questions/10532473/head-and-tail-in-one-line
   [3] https://stackoverflow.com/questions/646644/how-to-get-last-items-of-a-list-in-python
   [4] http://apache-spark-user-list.1001560.n3.nabble.com/RDD-tail-td4217.html
   [5] https://stackoverflow.com/questions/39544796/how-to-select-last-row-and-also-how-to-access-pyspark-dataframe-by-index
   [6] https://stackoverflow.com/questions/45406762/how-to-get-the-last-row-from-dataframe
   [7] https://issues.apache.org/jira/browse/SPARK-26433
   [8] https://github.com/databricks/koalas/issues/343
   [9] https://medium.com/@chris_bour/6-differences-between-pandas-and-spark-dataframes-1380cec394d2
   
   ### Does this PR introduce any user-facing change?
   
   No, (new API)
   
   ### How was this patch tested?
   
   Unit tests were added and manually tested.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org