You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by Alexander Smirnov <al...@gmail.com> on 2018/10/15 18:50:33 UTC

Joining streams

Hi everybody,

I'm migrating my program from Flink to Beam SDK and stuck on joining two
streams.

Flink has a dedicated operation "connect" and then you can apply CoFlatMap
function to handle messages from different streams.

Is there something similar in Beam?

Thank you,
Alex

Re: Joining streams

Posted by Csaba Kassai <cs...@aliz.ai>.

Hi Alexander,

I think what you are looking for in the Beam model is the PCollectionList
<https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/PCollectionList.html>
and
the Flatten
<https://beam.apache.org/releases/javadoc/2.7.0/index.html?org/apache/beam/sdk/values/PCollectionList.html>
transformation. Their usage is described here
<https://beam.apache.org/documentation/programming-guide/#flatten> and here
<https://cloud.google.com/dataflow/model/multiple-pcollections>:

Regards,
Csabi

On Tue, 16 Oct 2018 at 02:52, Kenneth Knowles <ke...@apache.org> wrote:

> Hi Alexander,
>
> In Beam you probably want CoGroupByKey [1]. You might also check out the
> Join Library [2].
>
> Kenn
>
> [1] https://beam.apache.org/documentation/programming-guide/#cogroupbykey
> [2]
> https://beam.apache.org/documentation/sdks/java-extensions/#join-library
>
> On Mon, Oct 15, 2018 at 11:50 AM Alexander Smirnov <
> alexander.smirnoff@gmail.com> wrote:
>
>> Hi everybody,
>>
>> I'm migrating my program from Flink to Beam SDK and stuck on joining two
>> streams.
>>
>> Flink has a dedicated operation "connect" and then you can apply
>> CoFlatMap function to handle messages from different streams.
>>
>> Is there something similar in Beam?
>>
>> Thank you,
>> Alex
>>
>

Re: Joining streams

Posted by Kenneth Knowles <ke...@apache.org>.

Hi Alexander,

In Beam you probably want CoGroupByKey [1]. You might also check out the
Join Library [2].

Kenn

[1] https://beam.apache.org/documentation/programming-guide/#cogroupbykey
[2] https://beam.apache.org/documentation/sdks/java-extensions/#join-library

On Mon, Oct 15, 2018 at 11:50 AM Alexander Smirnov <
alexander.smirnoff@gmail.com> wrote:

> Hi everybody,
>
> I'm migrating my program from Flink to Beam SDK and stuck on joining two
> streams.
>
> Flink has a dedicated operation "connect" and then you can apply CoFlatMap
> function to handle messages from different streams.
>
> Is there something similar in Beam?
>
> Thank you,
> Alex
>