You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@flink.apache.org by Deepak kumar Gunjetti <de...@gmail.com> on 2022/09/21 00:21:49 UTC
Is it possible to connect multiple streams
Hi,
My name is Deepak, I am a new user to apache flink. It is one of the best
open source i have used. I want to thank the community for developing such
a wonderful product.
I have one query.
Is it possible to connect multiple streams, like
stream1.connect(stream2).connect(stream3).flatmap(new
RickCoFlatMapFunctionHandler())
Can someone please let me know how I can achieve this.
Thanks,
Deepak
Re: Is it possible to connect multiple streams
Posted by Deepak kumar Gunjetti <de...@gmail.com>.
Thank you Gentlemen,
@yaroslav - I was very happy to see code on ppt. I checked on union and
join operators.
@shamoon - Thank you for the link I went through.
@Schwalbe - [Keyed]MultipleInputTransformation looks promising for me, as I
have more than 3 streams of different types and key type being the same. I
will check this further.
On Wed, Sep 21, 2022 at 4:03 PM Schwalbe Matthias <
Matthias.Schwalbe@viseca.ch> wrote:
> Hi Deepak,
>
>
>
> Coming back to your original question, you’ve got a number of option (some
> of them already mentioned:
>
> - You can connect/join 2 streams of *different* types at a time by
> means of s1.connect(s2).<some processing function>
> - (your example does not work directly as written (3 streams))
> - You can connect many streams of *same* type by means of s1.union(s2,
> s3, …).<some processing function>
> - The third option is new and not yet documented (marked as
> @experimental):
> - Connect/join many streams of different types by means of a
> [Keyed]MultipleInputTransformation
> - If keyed: input streams need to either
> - be keyed on the same key type, or
> - be non-keyed (most likely) broadcast streams
> - the API is still a little elaborate, a good starting point
> could be this test case: [1]
>
>
>
> If of any interest, feel free to ask for clarifications …
>
>
>
>
>
> Thias
>
> PS: see also my next email in a couple of minutes …
>
>
>
>
>
> [1]
> https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorBatchExecutionTest.java#L441
>
>
>
> *From:* Shammon FY <zj...@gmail.com>
> *Sent:* Wednesday, September 21, 2022 6:29 AM
> *To:* user@flink.apache.org
> *Cc:* Deepak kumar Gunjetti <de...@gmail.com>; Yaroslav Tkachenko <
> yaroslav@goldsky.com>
> *Subject:* Re: Is it possible to connect multiple streams
>
>
>
> ⚠*EXTERNAL MESSAGE – **CAUTION: Think Before You Click *⚠
>
>
>
> Hi
>
>
>
> Thanks @yaroslav .
>
> And @deepakgd79 here is the document for datastream:
> https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations
> You can find examples for union, connect, join and other transformations
>
>
>
>
>
> On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko <ya...@goldsky.com>
> wrote:
>
> Hi Deepak,
>
>
>
> You can use a union operator. I actually gave a talk on creating an
> advanced join using the union operator and multiple streams:
>
> -
> https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
>
> - https://www.youtube.com/watch?v=tiGxEGPyqCg
>
>
>
> I hope this helps.
>
>
>
> On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <
> deepakgb79@gmail.com> wrote:
>
> Hi,
>
> My name is Deepak, I am a new user to apache flink. It is one of the best
> open source i have used. I want to thank the community for developing such
> a wonderful product.
>
>
>
> I have one query.
>
> Is it possible to connect multiple streams, like
>
> stream1.connect(stream2).connect(stream3).flatmap(new
> RickCoFlatMapFunctionHandler())
>
>
>
> Can someone please let me know how I can achieve this.
>
> Thanks,
>
> Deepak
>
> Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und
> beinhaltet unter Umständen vertrauliche Mitteilungen. Da die
> Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann,
> übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und
> Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir
> Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie
> eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung
> dieser Informationen ist streng verboten.
>
> This message is intended only for the named recipient and may contain
> confidential or privileged information. As the confidentiality of email
> communication cannot be guaranteed, we do not accept any responsibility for
> the confidentiality and the intactness of this message. If you have
> received it in error, please advise the sender by return e-mail and delete
> this message and any attachments. Any unauthorised use or dissemination of
> this information is strictly prohibited.
>
RE: Is it possible to connect multiple streams
Posted by Schwalbe Matthias <Ma...@viseca.ch>.
Hi Deepak,
Coming back to your original question, you’ve got a number of option (some of them already mentioned:
* You can connect/join 2 streams of different types at a time by means of s1.connect(s2).<some processing function>
* (your example does not work directly as written (3 streams))
* You can connect many streams of same type by means of s1.union(s2, s3, …).<some processing function>
* The third option is new and not yet documented (marked as @experimental):
* Connect/join many streams of different types by means of a [Keyed]MultipleInputTransformation
* If keyed: input streams need to either
* be keyed on the same key type, or
* be non-keyed (most likely) broadcast streams
* the API is still a little elaborate, a good starting point could be this test case: [1]
If of any interest, feel free to ask for clarifications …
Thias
PS: see also my next email in a couple of minutes …
[1] https://github.com/apache/flink/blob/master/flink-streaming-java/src/test/java/org/apache/flink/streaming/api/graph/StreamGraphGeneratorBatchExecutionTest.java#L441
From: Shammon FY <zj...@gmail.com>
Sent: Wednesday, September 21, 2022 6:29 AM
To: user@flink.apache.org
Cc: Deepak kumar Gunjetti <de...@gmail.com>; Yaroslav Tkachenko <ya...@goldsky.com>
Subject: Re: Is it possible to connect multiple streams
⚠EXTERNAL MESSAGE – CAUTION: Think Before You Click ⚠
Hi
Thanks @yaroslav .
And @deepakgd79 here is the document for datastream: https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations You can find examples for union, connect, join and other transformations
On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko <ya...@goldsky.com>> wrote:
Hi Deepak,
You can use a union operator. I actually gave a talk on creating an advanced join using the union operator and multiple streams:
- https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
- https://www.youtube.com/watch?v=tiGxEGPyqCg
I hope this helps.
On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <de...@gmail.com>> wrote:
Hi,
My name is Deepak, I am a new user to apache flink. It is one of the best open source i have used. I want to thank the community for developing such a wonderful product.
I have one query.
Is it possible to connect multiple streams, like
stream1.connect(stream2).connect(stream3).flatmap(new RickCoFlatMapFunctionHandler())
Can someone please let me know how I can achieve this.
Thanks,
Deepak
Diese Nachricht ist ausschliesslich für den Adressaten bestimmt und beinhaltet unter Umständen vertrauliche Mitteilungen. Da die Vertraulichkeit von e-Mail-Nachrichten nicht gewährleistet werden kann, übernehmen wir keine Haftung für die Gewährung der Vertraulichkeit und Unversehrtheit dieser Mitteilung. Bei irrtümlicher Zustellung bitten wir Sie um Benachrichtigung per e-Mail und um Löschung dieser Nachricht sowie eventueller Anhänge. Jegliche unberechtigte Verwendung oder Verbreitung dieser Informationen ist streng verboten.
This message is intended only for the named recipient and may contain confidential or privileged information. As the confidentiality of email communication cannot be guaranteed, we do not accept any responsibility for the confidentiality and the intactness of this message. If you have received it in error, please advise the sender by return e-mail and delete this message and any attachments. Any unauthorised use or dissemination of this information is strictly prohibited.
Re: Is it possible to connect multiple streams
Posted by Shammon FY <zj...@gmail.com>.
Hi
Thanks @yaroslav .
And @deepakgd79 here is the document for datastream:
https://nightlies.apache.org/flink/flink-docs-master/docs/dev/datastream/operators/overview/#datastream-transformations
You can find examples for union, connect, join and other transformations
On Wed, Sep 21, 2022 at 11:55 AM Yaroslav Tkachenko <ya...@goldsky.com>
wrote:
> Hi Deepak,
>
> You can use a union operator. I actually gave a talk on creating an
> advanced join using the union operator and multiple streams:
> -
> https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
> - https://www.youtube.com/watch?v=tiGxEGPyqCg
>
> I hope this helps.
>
> On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <
> deepakgb79@gmail.com> wrote:
>
>> Hi,
>> My name is Deepak, I am a new user to apache flink. It is one of the best
>> open source i have used. I want to thank the community for developing such
>> a wonderful product.
>>
>> I have one query.
>> Is it possible to connect multiple streams, like
>> stream1.connect(stream2).connect(stream3).flatmap(new
>> RickCoFlatMapFunctionHandler())
>>
>> Can someone please let me know how I can achieve this.
>> Thanks,
>> Deepak
>>
>
Re: Is it possible to connect multiple streams
Posted by Yaroslav Tkachenko <ya...@goldsky.com>.
Hi Deepak,
You can use a union operator. I actually gave a talk on creating an
advanced join using the union operator and multiple streams:
-
https://www.slideshare.net/sap1ens/storing-state-forever-why-it-can-be-good-for-your-analytics
- https://www.youtube.com/watch?v=tiGxEGPyqCg
I hope this helps.
On Tue, Sep 20, 2022 at 5:22 PM Deepak kumar Gunjetti <de...@gmail.com>
wrote:
> Hi,
> My name is Deepak, I am a new user to apache flink. It is one of the best
> open source i have used. I want to thank the community for developing such
> a wonderful product.
>
> I have one query.
> Is it possible to connect multiple streams, like
> stream1.connect(stream2).connect(stream3).flatmap(new
> RickCoFlatMapFunctionHandler())
>
> Can someone please let me know how I can achieve this.
> Thanks,
> Deepak
>