group by optimizations with sorted input

[Richard Moorhead <ri...@gmail.com>]

In batch mode, if input is sorted prior to a group by operation; does flink
forward the aggregate data early? Is there a way to prevent grouping
operations from buffering all data in a GBK operation in batch mode?

Re: group by optimizations with sorted input

[Robert Metzger <rm...@apache.org>]

I assume you are using the DataSet API.
There, you can do a combinable group reduce:
https://ci.apache.org/projects/flink/flink-docs-master/dev/batch/dataset_transformations.html#combinable-groupreducefunctions

The combine() method will be executed on the sender side, reducing the
amount of data to spill on disk. This only works if your data allows such
early aggregations.
This is similar to a combiner in Hadoop:
https://www.quora.com/What-is-a-Combiner-in-Hadoop


On Thu, Feb 13, 2020 at 8:01 PM Richard Moorhead <ri...@gmail.com>
wrote:

> In batch mode, if input is sorted prior to a group by operation; does
> flink forward the aggregate data early? Is there a way to prevent grouping
> operations from buffering all data in a GBK operation in batch mode?
>