You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@flink.apache.org by Ufuk Celebi <uc...@apache.org> on 2015/06/19 13:04:17 UTC

Reduce combiner not chained

Hey all,

on the current master running the WordCount example with a text file input/output results and a manual reduce function (instead of the sum(1)) results in a combiner, which is not chained.

The corresponding issue is here: https://issues.apache.org/jira/browse/FLINK-2246

Can someone please confirm this? If it is an issue, we should fix it soon. The serialization overhead is noticeable on larger inputs.

– Ufuk

Re: Reduce combiner not chained

Posted by Ufuk Celebi <uc...@apache.org>.

I actually thought that this has been improved (not fixed ;)) some time ago. My mistake. Thanks for the update and checking again.

On 19 Jun 2015, at 13:47, Fabian Hueske <fh...@gmail.com> wrote:

> This is not a bug. Chained combiners are not supported for ReduceFunctions
> yet. :-(
> I updated the JIRA accordingly.
> 
> 2015-06-19 13:04 GMT+02:00 Ufuk Celebi <uc...@apache.org>:
> 
>> Hey all,
>> 
>> on the current master running the WordCount example with a text file
>> input/output results and a manual reduce function (instead of the sum(1))
>> results in a combiner, which is not chained.
>> 
>> The corresponding issue is here:
>> https://issues.apache.org/jira/browse/FLINK-2246
>> 
>> Can someone please confirm this? If it is an issue, we should fix it soon.
>> The serialization overhead is noticeable on larger inputs.
>> 
>> – Ufuk

Re: Reduce combiner not chained

Posted by Fabian Hueske <fh...@gmail.com>.

This is not a bug. Chained combiners are not supported for ReduceFunctions
yet. :-(
I updated the JIRA accordingly.

2015-06-19 13:04 GMT+02:00 Ufuk Celebi <uc...@apache.org>:

> Hey all,
>
> on the current master running the WordCount example with a text file
> input/output results and a manual reduce function (instead of the sum(1))
> results in a combiner, which is not chained.
>
> The corresponding issue is here:
> https://issues.apache.org/jira/browse/FLINK-2246
>
> Can someone please confirm this? If it is an issue, we should fix it soon.
> The serialization overhead is noticeable on larger inputs.
>
> – Ufuk