You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@flink.apache.org by Elias Levy <fe...@gmail.com> on 2017/08/19 18:02:19 UTC

Prioritize DataStream

I believe the answer to this question is "no", but I figure I might as well
ask.  Is there a way to prioritize a stream?

The use case is prioritization of a control stream.  This is mostly needed
on start-up, where a job might start consuming from the data stream before
consuming from the control stream.

Re: Prioritize DataStream

Posted by Till Rohrmann <tr...@apache.org>.

Hi Elias,

sorry for the slow answer. You were right that the answer is currently no.

However, people are currently working on changing the way the stream
operators work. This will allow the operator to decide from which input to
read next. Having such a functionality will enable us to implement proper
side inputs which will also cover your use case.

At the moment, one thing you could try is to buffer events in operator
state until you have fully consumed the other stream. Upon seeing all
events from the other stream (needs a proper termination event you can
detect) you can then replay all the buffered events. This could however
entail quite a big user state.

Cheers,
Till

On Mon, Aug 21, 2017 at 8:14 PM, Elias Levy <fe...@gmail.com>
wrote:

> Flink folks,
>
> A response to the question below?
>
> On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <fe...@gmail.com>
> wrote:
>
>> I believe the answer to this question is "no", but I figure I might as
>> well ask.  Is there a way to prioritize a stream?
>>
>> The use case is prioritization of a control stream.  This is mostly
>> needed on start-up, where a job might start consuming from the data stream
>> before consuming from the control stream.
>>
>>
>>
>

Re: Prioritize DataStream

Posted by Elias Levy <fe...@gmail.com>.

Flink folks,

A response to the question below?

On Sat, Aug 19, 2017 at 11:02 AM, Elias Levy <fe...@gmail.com>
wrote:

> I believe the answer to this question is "no", but I figure I might as
> well ask.  Is there a way to prioritize a stream?
>
> The use case is prioritization of a control stream.  This is mostly needed
> on start-up, where a job might start consuming from the data stream before
> consuming from the control stream.
>
>
>