You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by an0 <an...@gmail.com> on 2019/05/08 21:14:24 UTC

Rich and incrementally aggregating window functions

I want to use ProcessWindowFunction.Context#globalState in my window function. But I don't want to apply ProcessWindowFunction directly to my WindowedStream because I don't want to buffer all the elements of each window. Currently I'm using WindowedStream#aggregate(AggregateFunction, ProcessWindowFunction).

My understanding is that RichFunction.runtimeContext also give access to those global states. That thought naturally pointed me to RichAggregateFunction, RichReduceFunction and RichFoldFunction. However, they all cause runtime error like this:
"AggregateFunction can not be a RichFunction. Please use fold(AggregateFunction, WindowFunction) instead."

So how can I use an incrementally aggregating window function and have access to global states at the same time?

Re: Rich and incrementally aggregating window functions

Posted by an0 <an...@gmail.com>.

Thanks. I know reimplementing windowing myself will work but that's a very bad last resort.

I believe it is a very reasonable request. But since someone else has already filed a Jira and it was closed as Won't Fix[1], I won't bother refiling it again. I'll try something else first. 

[1]
https://issues.apache.org/jira/browse/FLINK-10250

On 2019/05/09 02:07:00, Hequn Cheng <ch...@gmail.com> wrote: 
> Hi,
> 
> There is a discussion about this before, you can take a look at it[1].
> Best, Hequn
> 
> [1]
> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-implementation-of-aggregate-function-using-a-ProcessFunction-td23473.html#a23531
> 
> On Thu, May 9, 2019 at 5:14 AM an0 <an...@gmail.com> wrote:
> 
> > I want to use ProcessWindowFunction.Context#globalState in my window
> > function. But I don't want to apply ProcessWindowFunction directly to my
> > WindowedStream because I don't want to buffer all the elements of each
> > window. Currently I'm using WindowedStream#aggregate(AggregateFunction,
> > ProcessWindowFunction).
> >
> > My understanding is that RichFunction.runtimeContext also give access to
> > those global states. That thought naturally pointed me to
> > RichAggregateFunction, RichReduceFunction and RichFoldFunction. However,
> > they all cause runtime error like this:
> > "AggregateFunction can not be a RichFunction. Please use
> > fold(AggregateFunction, WindowFunction) instead."
> >
> > So how can I use an incrementally aggregating window function and have
> > access to global states at the same time?
> >
>

Re: Rich and incrementally aggregating window functions

Posted by Hequn Cheng <ch...@gmail.com>.

Hi,

There is a discussion about this before, you can take a look at it[1].
Best, Hequn

[1]
http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/Regarding-implementation-of-aggregate-function-using-a-ProcessFunction-td23473.html#a23531

On Thu, May 9, 2019 at 5:14 AM an0 <an...@gmail.com> wrote:

> I want to use ProcessWindowFunction.Context#globalState in my window
> function. But I don't want to apply ProcessWindowFunction directly to my
> WindowedStream because I don't want to buffer all the elements of each
> window. Currently I'm using WindowedStream#aggregate(AggregateFunction,
> ProcessWindowFunction).
>
> My understanding is that RichFunction.runtimeContext also give access to
> those global states. That thought naturally pointed me to
> RichAggregateFunction, RichReduceFunction and RichFoldFunction. However,
> they all cause runtime error like this:
> "AggregateFunction can not be a RichFunction. Please use
> fold(AggregateFunction, WindowFunction) instead."
>
> So how can I use an incrementally aggregating window function and have
> access to global states at the same time?
>