You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/17 15:53:34 UTC
[GitHub] [spark] EnricoMi opened a new pull request #35899: Add flatMapSortedGroups to KeyValueGroupedDataset
EnricoMi opened a new pull request #35899:
URL: https://github.com/apache/spark/pull/35899
### What changes were proposed in this pull request?
This adds a sorted version of `ds.groupByKey(…).flatMapGroups(…)`.
### Why are the changes needed?
The existing method `flatMapGroups` provides an iterator of rows for each group key. If user code would requires those rows in a particular order, that iterator would have to be sorted first, which is against the idea of an iterator in the first place. For groups that do not fit into memory of one executor, this approach does not work.
[org.apache.spark.sql.KeyValueGroupedDataset](https://github.com/apache/spark/blob/47485a3c2df3201c838b939e82d5b26332e2d858/sql/core/src/main/scala/org/apache/spark/sql/KeyValueGroupedDataset.scala#L134-L137):
> Internally, the implementation will spill to disk if any given group is too large to fit into
> memory. However, users must take care to avoid materializing the whole iterator for a group
> (for example, by calling `toList`) unless they are sure that this is possible given the memory
> constraints of their cluster.
The implementation of `KeyValueGroupedDataset.flatMapGroups` already sorts each partition according to the group key. By additionally sorting by some data columns, the iterator can be guaranteed to provide some order.
### Does this PR introduce _any_ user-facing change?
This adds `KeyValueGroupedDataset.flatMapSortedGroups`.
### How was this patch tested?
There is test `DatasetSuite."groupBy function, flatMapSorted by func"` and `DatasetSuite."groupBy function, flatMapSorted by expr"`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #35899: [SPARK-38591][SQL] Add flatMapSortedGroups to KeyValueGroupedDataset
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #35899:
URL: https://github.com/apache/spark/pull/35899#issuecomment-1073022317
Can one of the admins verify this patch?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org