You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flink.apache.org by 魏偉哲 <to...@gmail.com> on 2017/04/26 04:39:09 UTC
Question on window ProcessFunction
Hi all,
Recently, I was tracing the source code in streaming api and I was confused
about some implementations.
When using reduce function with evictor, the *WindowStream* will wrap the
*ReduceFunction* and *ProcessWindowFunction* into
*ReduceApplyProcessWindonwFunction* and put it in
*InternalIterableProcessWindowFunction*. So does fold function.
However, when using aggregate, the *InternalIterableProcessWindowFunction*
was changed to *InternalAggregateProcessWindowFunction* which was applied
aggregation in the process() method.
My question is why not implement an *AggregateApplyProcessWindowFunction*
and use *InternalIterableProcessWindowFunction* instead just like reduce,
fold function did. Is there any concern?
Many thanks,
Tony Wei
Re: Question on window ProcessFunction
Posted by 魏偉哲 <to...@gmail.com>.
Hi Aljoscha,
I see. Thanks for your reply.
Best,
Tony Wei
Aljoscha Krettek <al...@apache.org>於 2017年4月26日 週三，下午10:29寫道：
> Hi,
> Both implementations work so no one bothered to change the older
> implementations yet. I don’t think it’s a problem but if you want you can
> adapt reduce/fold to the newer implementation.
>
> Best,
> Aljoscha
> > On 26. Apr 2017, at 14:51, 魏偉哲 <to...@gmail.com> wrote:
> >
> > Hi Aljoscha,
> >
> > I know the aggregate code is newer. I am confused because the
> > implementations are not consistent.
> > Does it mean that the reduce/fold implementation would need to be
> > refactored for the purpose of having less layers ?
> > Or is it better to remain the current implementations for some reasons?
> >
> > Many thanks,
> > Tony Wei
> >
> > 2017-04-26 20:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
> >
> >> Hi Tony,
> >> The reason for this is that the aggregate code is newer. The new code
> has
> >> less layers, compared to the reduce/fold implementation where it is
> >> InternalFunction(ReduceApplyFunction(Reduce)) instead of
> >> InteralAggregateFunction(Aggregate).
> >>
> >> Best,
> >> Aljoscha
> >>> On 26. Apr 2017, at 06:39, 魏偉哲 <to...@gmail.com> wrote:
> >>>
> >>> Hi all,
> >>>
> >>> Recently, I was tracing the source code in streaming api and I was
> >> confused
> >>> about some implementations.
> >>>
> >>> When using reduce function with evictor, the *WindowStream* will wrap
> the
> >>> *ReduceFunction* and *ProcessWindowFunction* into
> >>> *ReduceApplyProcessWindonwFunction* and put it in
> >>> *InternalIterableProcessWindowFunction*. So does fold function.
> >>>
> >>> However, when using aggregate, the *InternalIterableProcessWindowF
> >> unction*
> >>> was changed to *InternalAggregateProcessWindowFunction* which was
> >> applied
> >>> aggregation in the process() method.
> >>>
> >>> My question is why not implement an *AggregateApplyProcessWindowFun
> >> ction*
> >>> and use *InternalIterableProcessWindowFunction* instead just like
> >> reduce,
> >>> fold function did. Is there any concern?
> >>>
> >>> Many thanks,
> >>> Tony Wei
> >>
> >>
>
>
Re: Question on window ProcessFunction
Posted by Aljoscha Krettek <al...@apache.org>.
Hi,
Both implementations work so no one bothered to change the older implementations yet. I don’t think it’s a problem but if you want you can adapt reduce/fold to the newer implementation.
Best,
Aljoscha
> On 26. Apr 2017, at 14:51, 魏偉哲 <to...@gmail.com> wrote:
>
> Hi Aljoscha,
>
> I know the aggregate code is newer. I am confused because the
> implementations are not consistent.
> Does it mean that the reduce/fold implementation would need to be
> refactored for the purpose of having less layers ?
> Or is it better to remain the current implementations for some reasons?
>
> Many thanks,
> Tony Wei
>
> 2017-04-26 20:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
>
>> Hi Tony,
>> The reason for this is that the aggregate code is newer. The new code has
>> less layers, compared to the reduce/fold implementation where it is
>> InternalFunction(ReduceApplyFunction(Reduce)) instead of
>> InteralAggregateFunction(Aggregate).
>>
>> Best,
>> Aljoscha
>>> On 26. Apr 2017, at 06:39, 魏偉哲 <to...@gmail.com> wrote:
>>>
>>> Hi all,
>>>
>>> Recently, I was tracing the source code in streaming api and I was
>> confused
>>> about some implementations.
>>>
>>> When using reduce function with evictor, the *WindowStream* will wrap the
>>> *ReduceFunction* and *ProcessWindowFunction* into
>>> *ReduceApplyProcessWindonwFunction* and put it in
>>> *InternalIterableProcessWindowFunction*. So does fold function.
>>>
>>> However, when using aggregate, the *InternalIterableProcessWindowF
>> unction*
>>> was changed to *InternalAggregateProcessWindowFunction* which was
>> applied
>>> aggregation in the process() method.
>>>
>>> My question is why not implement an *AggregateApplyProcessWindowFun
>> ction*
>>> and use *InternalIterableProcessWindowFunction* instead just like
>> reduce,
>>> fold function did. Is there any concern?
>>>
>>> Many thanks,
>>> Tony Wei
>>
>>
Re: Question on window ProcessFunction
Posted by 魏偉哲 <to...@gmail.com>.
Hi Aljoscha,
I know the aggregate code is newer. I am confused because the
implementations are not consistent.
Does it mean that the reduce/fold implementation would need to be
refactored for the purpose of having less layers ?
Or is it better to remain the current implementations for some reasons?
Many thanks,
Tony Wei
2017-04-26 20:24 GMT+08:00 Aljoscha Krettek <al...@apache.org>:
> Hi Tony,
> The reason for this is that the aggregate code is newer. The new code has
> less layers, compared to the reduce/fold implementation where it is
> InternalFunction(ReduceApplyFunction(Reduce)) instead of
> InteralAggregateFunction(Aggregate).
>
> Best,
> Aljoscha
> > On 26. Apr 2017, at 06:39, 魏偉哲 <to...@gmail.com> wrote:
> >
> > Hi all,
> >
> > Recently, I was tracing the source code in streaming api and I was
> confused
> > about some implementations.
> >
> > When using reduce function with evictor, the *WindowStream* will wrap the
> > *ReduceFunction* and *ProcessWindowFunction* into
> > *ReduceApplyProcessWindonwFunction* and put it in
> > *InternalIterableProcessWindowFunction*. So does fold function.
> >
> > However, when using aggregate, the *InternalIterableProcessWindowF
> unction*
> > was changed to *InternalAggregateProcessWindowFunction* which was
> applied
> > aggregation in the process() method.
> >
> > My question is why not implement an *AggregateApplyProcessWindowFun
> ction*
> > and use *InternalIterableProcessWindowFunction* instead just like
> reduce,
> > fold function did. Is there any concern?
> >
> > Many thanks,
> > Tony Wei
>
>
Re: Question on window ProcessFunction
Posted by Aljoscha Krettek <al...@apache.org>.
Hi Tony,
The reason for this is that the aggregate code is newer. The new code has less layers, compared to the reduce/fold implementation where it is InternalFunction(ReduceApplyFunction(Reduce)) instead of InteralAggregateFunction(Aggregate).
Best,
Aljoscha
> On 26. Apr 2017, at 06:39, 魏偉哲 <to...@gmail.com> wrote:
>
> Hi all,
>
> Recently, I was tracing the source code in streaming api and I was confused
> about some implementations.
>
> When using reduce function with evictor, the *WindowStream* will wrap the
> *ReduceFunction* and *ProcessWindowFunction* into
> *ReduceApplyProcessWindonwFunction* and put it in
> *InternalIterableProcessWindowFunction*. So does fold function.
>
> However, when using aggregate, the *InternalIterableProcessWindowFunction*
> was changed to *InternalAggregateProcessWindowFunction* which was applied
> aggregation in the process() method.
>
> My question is why not implement an *AggregateApplyProcessWindowFunction*
> and use *InternalIterableProcessWindowFunction* instead just like reduce,
> fold function did. Is there any concern?
>
> Many thanks,
> Tony Wei