You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Sigalit Eliazov <e....@gmail.com> on 2022/05/10 10:09:11 UTC

sharing data between 2 pipelines

Hi all
i have 2 pipelines:
A. receives information from kafka and "holds" that info
B. a pipeline which is triggered by a scheduler and every x minutes should
send the info i received in pipeline A to another kafka topic

As i understood i cannot use the flink state for this since these are
different pipelines/operators..
is there a way to implement such a case in Flink itself without using any
external application like redis or db?

Thanks
Sigalit

Re: sharing data between 2 pipelines

Posted by yidan zhao <hi...@gmail.com>.
why not put the two pipelines together.

David Anderson <da...@apache.org> 于2022年5月11日周三 00:13写道:
>
> This sounds like it might be a use case for something like a KeyedCoProcessFunction (or possibly a KeyedBroadcastProcessFunction, depending on the details). These operators can receive inputs from two different sources, and share state between them.
>
> The rides and fares exercise [1] from the flink-training illustrates this pattern, which you can also read about in the tutorial on Connected Streams in the documentation [2].
>
> If you need to broadcast the signal from the scheduler, see [3].
>
> Regards,
> David
>
> [1] https://github.com/apache/flink-training/tree/master/rides-and-fares
> [2] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/learn-flink/etl/#connected-streams
> [3] https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/broadcast_state/
>
> On Tue, May 10, 2022 at 12:09 PM Sigalit Eliazov <e....@gmail.com> wrote:
>>
>> Hi all
>> i have 2 pipelines:
>> A. receives information from kafka and "holds" that info
>> B. a pipeline which is triggered by a scheduler and every x minutes should send the info i received in pipeline A to another kafka topic
>>
>> As i understood i cannot use the flink state for this since these are different pipelines/operators..
>> is there a way to implement such a case in Flink itself without using any external application like redis or db?
>>
>> Thanks
>> Sigalit
>>

Re: sharing data between 2 pipelines

Posted by David Anderson <da...@apache.org>.
This sounds like it might be a use case for something like a
KeyedCoProcessFunction (or possibly a KeyedBroadcastProcessFunction,
depending on the details). These operators can receive inputs from two
different sources, and share state between them.

The rides and fares exercise [1] from the flink-training illustrates this
pattern, which you can also read about in the tutorial on Connected Streams
in the documentation [2].

If you need to broadcast the signal from the scheduler, see [3].

Regards,
David

[1] https://github.com/apache/flink-training/tree/master/rides-and-fares
[2]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/learn-flink/etl/#connected-streams
[3]
https://nightlies.apache.org/flink/flink-docs-release-1.15/docs/dev/datastream/fault-tolerance/broadcast_state/

On Tue, May 10, 2022 at 12:09 PM Sigalit Eliazov <e....@gmail.com>
wrote:

> Hi all
> i have 2 pipelines:
> A. receives information from kafka and "holds" that info
> B. a pipeline which is triggered by a scheduler and every x minutes should
> send the info i received in pipeline A to another kafka topic
>
> As i understood i cannot use the flink state for this since these are
> different pipelines/operators..
> is there a way to implement such a case in Flink itself without using any
> external application like redis or db?
>
> Thanks
> Sigalit
>
>