You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by Viktor Rosenfeld <vi...@tu-berlin.de> on 2014/11/21 13:05:50 UTC
Translate one Java API operator to multiple common API operators?
Hi,
Is it possible to implement one Java API operator using multiple Common API
operators?
I.e., turn something like
IN -> OP(aggregate) -> OUT
into
IN -> Map1 -> Reduce -> Map2 -> OUT
I would imagine that the method SingleInputOperator.translateToDataFlow
could just create a chain of operators, wire the input to the first operator
(Map1) and return the last operator of the chain (Map2).
Is there anything I need to be aware of with this approach, e.g., some
performance issues?
Best,
Viktor
--
View this message in context: http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
Sent from the Apache Flink (Incubator) Mailing List archive. mailing list archive at Nabble.com.
Re: Translate one Java API operator to multiple common API operators?
Posted by Fabian Hueske <fh...@apache.org>.
correction, there is no map(unwrap) in the example I gave.
2014-11-21 13:15 GMT+01:00 Fabian Hueske <fh...@apache.org>:
> This is possible and already done.
> For example, a key-selector-groupReduce is translated into sth like:
> map(keySelector)->groupReduce()->map(unwrap)
>
> Since, MapFunctions are quite lightweight, there shouldn't be major
> performance implications. Make sure you get the DOP of all operators right,
> though.
>
> 2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <viktor.rosenfeld@tu-berlin.de
> >:
>
>> Hi,
>>
>> Is it possible to implement one Java API operator using multiple Common
>> API
>> operators?
>>
>> I.e., turn something like
>>
>> IN -> OP(aggregate) -> OUT
>>
>> into
>>
>> IN -> Map1 -> Reduce -> Map2 -> OUT
>>
>> I would imagine that the method SingleInputOperator.translateToDataFlow
>> could just create a chain of operators, wire the input to the first
>> operator
>> (Map1) and return the last operator of the chain (Map2).
>>
>> Is there anything I need to be aware of with this approach, e.g., some
>> performance issues?
>>
>> Best,
>> Viktor
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
>> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
>> archive at Nabble.com.
>>
>
>
Re: Translate one Java API operator to multiple common API operators?
Posted by Fabian Hueske <fh...@apache.org>.
This is possible and already done.
For example, a key-selector-groupReduce is translated into sth like:
map(keySelector)->groupReduce()->map(unwrap)
Since, MapFunctions are quite lightweight, there shouldn't be major
performance implications. Make sure you get the DOP of all operators right,
though.
2014-11-21 13:05 GMT+01:00 Viktor Rosenfeld <vi...@tu-berlin.de>:
> Hi,
>
> Is it possible to implement one Java API operator using multiple Common API
> operators?
>
> I.e., turn something like
>
> IN -> OP(aggregate) -> OUT
>
> into
>
> IN -> Map1 -> Reduce -> Map2 -> OUT
>
> I would imagine that the method SingleInputOperator.translateToDataFlow
> could just create a chain of operators, wire the input to the first
> operator
> (Map1) and return the last operator of the chain (Map2).
>
> Is there anything I need to be aware of with this approach, e.g., some
> performance issues?
>
> Best,
> Viktor
>
>
>
> --
> View this message in context:
> http://apache-flink-incubator-mailing-list-archive.1008284.n3.nabble.com/Translate-one-Java-API-operator-to-multiple-common-API-operators-tp2567.html
> Sent from the Apache Flink (Incubator) Mailing List archive. mailing list
> archive at Nabble.com.
>