You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by tao xiao <xi...@gmail.com> on 2017/01/07 12:30:43 UTC
window function outputs two different values
Hi team,
I have a requirement that wants to output two different values from a time
window reduce function. Here is basic workflow
1. fetch data from Kafka
2. flow the data to a event session window. kafka source -> keyBy ->
session window -> reduce
3. inside the reduce function, count the number of data and also emit the
data itself to another operator for further processing
As the reduce function can only emit the count, I want to know how to also
emit the data as well?
Re: window function outputs two different values
Posted by Yury Ruchin <yu...@gmail.com>.
Hi,
Is there a strict requirement that elements must proceed along the
processing pipeline exactly after being accounted by the reduce function?
If not, you could derive two streams from the original one to be processed
concurrently, something like this:
val protoStream = kafka source -> keyBy
val aggregateStream = protoStream -> window -> reduce
val someOtherStream = protoStream -> <other processing operators go here>
Or, if the above is not an option and window collection latency is not an
issue, you could just use generic window function or fold function. The
former gives access to window elements as an iterable, the latter allows
using custom accumulator that contains the intermediate count and window
elements seen so far.
Regards,
Yury
2017-01-10 17:43 GMT+03:00 Aljoscha Krettek <al...@apache.org>:
> Hi,
> I'm afraid this is not possible with the current model. A reduce function
> is only meant to combine two values and output the result of that. Side
> effects, such as emitting further data are not allowed right now.
>
> Cheers,
> Aljoscha
>
> On Mon, 9 Jan 2017 at 15:27 tao xiao <xi...@gmail.com> wrote:
>
>> Hi team,
>>
>> any suggestions on below topic?
>>
>> I have a requirement that wants to output two different values from a
>> time window reduce function. Here is basic workflow
>>
>> 1. fetch data from Kafka
>> 2. flow the data to a event session window. kafka source -> keyBy ->
>> session window -> reduce
>> 3. inside the reduce function, count the number of data and also emit the
>> data itself to another operator for further processing
>>
>> As the reduce function can only emit the count, I want to know how to
>> also emit the data as well?
>>
>>
>>
>> On Sat, 7 Jan 2017 at 20:30 tao xiao <xi...@gmail.com> wrote:
>>
>> Hi team,
>>
>> I have a requirement that wants to output two different values from a
>> time window reduce function. Here is basic workflow
>>
>> 1. fetch data from Kafka
>> 2. flow the data to a event session window. kafka source -> keyBy ->
>> session window -> reduce
>> 3. inside the reduce function, count the number of data and also emit the
>> data itself to another operator for further processing
>>
>> As the reduce function can only emit the count, I want to know how to
>> also emit the data as well?
>>
>>
Re: window function outputs two different values
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
I'm afraid this is not possible with the current model. A reduce function
is only meant to combine two values and output the result of that. Side
effects, such as emitting further data are not allowed right now.
Cheers,
Aljoscha
On Mon, 9 Jan 2017 at 15:27 tao xiao <xi...@gmail.com> wrote:
> Hi team,
>
> any suggestions on below topic?
>
> I have a requirement that wants to output two different values from a time
> window reduce function. Here is basic workflow
>
> 1. fetch data from Kafka
> 2. flow the data to a event session window. kafka source -> keyBy ->
> session window -> reduce
> 3. inside the reduce function, count the number of data and also emit the
> data itself to another operator for further processing
>
> As the reduce function can only emit the count, I want to know how to also
> emit the data as well?
>
>
>
> On Sat, 7 Jan 2017 at 20:30 tao xiao <xi...@gmail.com> wrote:
>
> Hi team,
>
> I have a requirement that wants to output two different values from a time
> window reduce function. Here is basic workflow
>
> 1. fetch data from Kafka
> 2. flow the data to a event session window. kafka source -> keyBy ->
> session window -> reduce
> 3. inside the reduce function, count the number of data and also emit the
> data itself to another operator for further processing
>
> As the reduce function can only emit the count, I want to know how to also
> emit the data as well?
>
>
Re: window function outputs two different values
Posted by tao xiao <xi...@gmail.com>.
Hi team,
any suggestions on below topic?
I have a requirement that wants to output two different values from a time
window reduce function. Here is basic workflow
1. fetch data from Kafka
2. flow the data to a event session window. kafka source -> keyBy ->
session window -> reduce
3. inside the reduce function, count the number of data and also emit the
data itself to another operator for further processing
As the reduce function can only emit the count, I want to know how to also
emit the data as well?
On Sat, 7 Jan 2017 at 20:30 tao xiao <xi...@gmail.com> wrote:
> Hi team,
>
> I have a requirement that wants to output two different values from a time
> window reduce function. Here is basic workflow
>
> 1. fetch data from Kafka
> 2. flow the data to a event session window. kafka source -> keyBy ->
> session window -> reduce
> 3. inside the reduce function, count the number of data and also emit the
> data itself to another operator for further processing
>
> As the reduce function can only emit the count, I want to know how to also
> emit the data as well?
>
>