You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by nragon <nu...@wedotechnologies.com> on 2017/06/12 16:25:50 UTC

ReduceFunction mechanism

Hi,

Regarding ReduceFunction.
Is reduce() called when there is only one record for a given key?

Thanks



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: ReduceFunction mechanism

Posted by nragon <nu...@wedotechnologies.com>.
just an fyi: currently i'm doing map -> keyby -> reduce which in fact could
only be keyby -> reduce since reduce can have the map logic.



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651p13740.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: ReduceFunction mechanism

Posted by nragon <nu...@wedotechnologies.com>.
My goal is to touch each element before the aggregation but i could do it in
the reduce function a not having to add another function, thus creating more
overhead. The reduce method receives the reduced and a new element which i
would change and apply my aggregation.
I'm doing keyby->reduce.
Using a map before all this is a solution.
I've  never tried AggregateFunction, any examples?

Thanks,
Nuno



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651p13723.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: ReduceFunction mechanism

Posted by Fabian Hueske <fh...@gmail.com>.
You can use a MapFunction (however, it will touch each element and not only
the first).
An alternative could be the AggregateFunction if you are using
ReduceFunction on a WindowedStream. The interface is a bit more complex
though.

Best, Fabian



2017-06-13 10:55 GMT+02:00 nragon <nu...@wedotechnologies.com>:

> So, if my reduce function applies some transformation I must migrate that
> transformation to a map before the reduce to ensure it transforms, even if
> there is only one element?
> I can chain them together and it will be "almost" as they were in the same
> function(Ensure same thread processing)?
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-
> mechanism-tp13651p13679.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>

Re: ReduceFunction mechanism

Posted by nragon <nu...@wedotechnologies.com>.
So, if my reduce function applies some transformation I must migrate that
transformation to a map before the reduce to ensure it transforms, even if
there is only one element?
I can chain them together and it will be "almost" as they were in the same
function(Ensure same thread processing)?



--
View this message in context: http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-mechanism-tp13651p13679.html
Sent from the Apache Flink User Mailing List archive. mailing list archive at Nabble.com.

Re: ReduceFunction mechanism

Posted by Fabian Hueske <fh...@gmail.com>.
No, the reduce() method of a ReduceFunction requires two elements.
The first received element is just put into state. Once the second element
arrives, both are given to the ReduceFunction and the result is put into
state and replaces the first element.

Best, Fabian

2017-06-12 18:25 GMT+02:00 nragon <nu...@wedotechnologies.com>:

> Hi,
>
> Regarding ReduceFunction.
> Is reduce() called when there is only one record for a given key?
>
> Thanks
>
>
>
> --
> View this message in context: http://apache-flink-user-
> mailing-list-archive.2336050.n4.nabble.com/ReduceFunction-
> mechanism-tp13651.html
> Sent from the Apache Flink User Mailing List archive. mailing list archive
> at Nabble.com.
>