Joining two streams of different priorities

["Aggarwal, Ajay" <Aj...@netapp.com>]

My main input stream (inputStream1) gets processed using a pipeline that looks like below

                                inputStream1
                                .keyBy("some-key")
                                .window(TumblingEventTimeWindows.of(Time.seconds(Properties.WINDOW_SIZE)))
                                .process(new MyProcessWindowFunction());


Note that for each key only place I want main processing to happen is in the above ProcessWindowFunction. That’s because I have a requirement to process events associated with a key serially (one at a time) and in order by eventTime.

However I have 2 sources of events that I need to join:

  *   inputStream1 : normal priority events (using eventTime)
  *   inputStream2:  higher priority events

I somehow need to join these 2 streams, so all processing per key is still happening in same ProcessWindowFunction. But I want to treat events from inputStream2 with higher priority. So if pipeline is backed up a little with events from inputStream1  and a new event shows up in inputStream2, I want that event to make it to ProcessWindowFunction faster.  Is there any way to make that happen? Window Join will not give me the desired behavior since it will join elements of two streams that lie in same window.

Thanks!

Ajay

Re: Joining two streams of different priorities

[Fabian Hueske <fh...@gmail.com>]

Hi,

This is not possible with Flink. Events in transport channels cannot be
reordered and function cannot pick which input to read from.
There are some upcoming changes for the unified batch-stream integration
that enable to chose which input to read from, but this is not there yet,
AFAIK.

Best, Fabian

Am Mi., 6. März 2019 um 16:16 Uhr schrieb Aggarwal, Ajay <
Ajay.Aggarwal@netapp.com>:

> My main input stream (inputStream1) gets processed using a pipeline that
> looks like below
>
>
>
>                                 inputStream1
>
>                                 .keyBy("some-key")
>
>
> .window(TumblingEventTimeWindows.of(Time.seconds(Properties.WINDOW_SIZE)))
>
>                                 .process(new MyProcessWindowFunction());
>
>
>
>
>
> Note that for each key only place I want main processing to happen is in
> the above ProcessWindowFunction. That’s because I have a requirement to
> process events associated with a key serially (one at a time) and in order
> by eventTime.
>
>
>
> However I have 2 sources of events that I need to join:
>
>    - inputStream1 : normal priority events (using eventTime)
>    - inputStream2:  higher priority events
>
>
>
> I somehow need to join these 2 streams, so all processing per key is still
> happening in same ProcessWindowFunction. But I want to treat events from
> inputStream2 with higher priority. So if pipeline is backed up a little
> with events from inputStream1  and a new event shows up in inputStream2, I
> want that event to make it to ProcessWindowFunction faster.  Is there any
> way to make that happen? Window Join will not give me the desired behavior
> since it will join elements of two streams that lie in same window.
>
>
>
> Thanks!
>
>
>
> Ajay
>
>
>
>
>