You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by "LINZ, Arnaud" <AL...@bouyguestelecom.fr> on 2020/01/30 08:17:52 UTC
How to fully re-aggregate a keyed windowed aggregate in the same
window ?
Hello,
I would like to compute statistics on a stream every hour. For that, I need to compute statistics on the keyed stream, then to reaggregate them.
I’ve tried the following thing :
stream.keyBy(mykey)
.window(1 hour process time)
.aggregate(my per-key aggregate)
.windowAll(1 hour process time) // not the same window, add one hour delay…
.reduce(fully aggregate intermediary results)
... then sink
This works, but I get the first line in the sink 2 hours after the first item in the sink, and 1 hour after it should be possible to get it.
My question: How to I trigger the reduce step immediately after the first aggregation ?
Best regards,
Arnaud
________________________________
L'intégrité de ce message n'étant pas assurée sur internet, la société expéditrice ne peut être tenue responsable de son contenu ni de ses pièces jointes. Toute utilisation ou diffusion non autorisée est interdite. Si vous n'êtes pas destinataire de ce message, merci de le détruire et d'avertir l'expéditeur.
The integrity of this message cannot be guaranteed on the Internet. The company that sent this message cannot therefore be held liable for its content nor attachments. Any unauthorized use or dissemination is prohibited. If you are not the intended recipient of this message, then please delete it and notify the sender.
Re: How to fully re-aggregate a keyed windowed aggregate in the same
window ?
Posted by Robert Metzger <rm...@apache.org>.
Hey Arnaud,
sorry that you didn't get an answer yet. Were you able to solve your
problem in the meantime? If not, I'll find somebody to answer your question
:)
On Thu, Jan 30, 2020 at 9:18 AM LINZ, Arnaud <AL...@bouyguestelecom.fr>
wrote:
> Hello,
>
>
>
> I would like to compute statistics on a stream every hour. For that, I
> need to compute statistics on the keyed stream, then to reaggregate them.
>
> I’ve tried the following thing :
>
>
>
> stream.keyBy(mykey)
>
> .window(1 hour process time)
>
> .aggregate(my per-key aggregate)
>
>
>
> .windowAll(1 hour process time) // not the same window, add
> one hour delay…
>
>
>
> .reduce(fully aggregate intermediary results)
>
> ... then sink
>
>
>
> This works, but I get the first line in the sink 2 hours after the first
> item in the sink, and 1 hour after it should be possible to get it.
>
>
>
> My question: How to I trigger the reduce step immediately after the first
> aggregation ?
>
>
>
> Best regards,
>
> Arnaud
>
>
>
>
>
> ------------------------------
>
> L'intégrité de ce message n'étant pas assurée sur internet, la société
> expéditrice ne peut être tenue responsable de son contenu ni de ses pièces
> jointes. Toute utilisation ou diffusion non autorisée est interdite. Si
> vous n'êtes pas destinataire de ce message, merci de le détruire et
> d'avertir l'expéditeur.
>
> The integrity of this message cannot be guaranteed on the Internet. The
> company that sent this message cannot therefore be held liable for its
> content nor attachments. Any unauthorized use or dissemination is
> prohibited. If you are not the intended recipient of this message, then
> please delete it and notify the sender.
>