Sorting in a WindowedDataStream

[Niklas Semmler <ni...@inet.tu-berlin.de>]

Hello there,

What functions should be used to aggregate (unordered) tuples for every 
window in a WindowedDataStream to a (ordered) list?

Neither foldWindow nor reduceWindow seems to be applicable, and 
aggregate does not, to my understanding, take user-defined functions.

To get started with flink I am experimenting with the 
WindowedDataStream. My goal is to naively compute the top-k from a 
stream of key-value tuples.

From: Input over SocketTextStream
previous window, (a, 1), (b, 3), (c, 2), next window

To:
[previous window], [(b, 3), (c, 2), (a, 1)], [next window]

Is this behavior not possible?

Best,
Niklas

-- 
PhD Student / Research Assistant
INET, TU Berlin
Room 4.029
Marchstr 23
10587 Berlin
Tel: +49 30 314 78752

Re: Sorting in a WindowedDataStream

[Márton Balassi <ba...@gmail.com>]

Dear Niklas,

To do that you can use WindowedDataStream.mapWindow(). This gives you an
iterator to all the records in the window and you can do whatever you wish
with them.

One thing to note if sorting windows of the stream might add considerable
latency to your job.

Best,

Marton

On Tue, Apr 14, 2015 at 12:12 PM, Niklas Semmler <ni...@inet.tu-berlin.de>
wrote:

> Hello there,
>
> What functions should be used to aggregate (unordered) tuples for every
> window in a WindowedDataStream to a (ordered) list?
>
> Neither foldWindow nor reduceWindow seems to be applicable, and aggregate
> does not, to my understanding, take user-defined functions.
>
> To get started with flink I am experimenting with the WindowedDataStream.
> My goal is to naively compute the top-k from a stream of key-value tuples.
>
> From: Input over SocketTextStream
> previous window, (a, 1), (b, 3), (c, 2), next window
>
> To:
> [previous window], [(b, 3), (c, 2), (a, 1)], [next window]
>
> Is this behavior not possible?
>
> Best,
> Niklas
>
> --
> PhD Student / Research Assistant
> INET, TU Berlin
> Room 4.029
> Marchstr 23
> 10587 Berlin
> Tel: +49 30 314 78752
>