Need Help

[Pramod Kumar <de...@gmail.com>]

Hi there,
Hope everyone is doing well.
I am stuck at a problem where I have to merge two pcollections one of it is
pretty huge.
While merging both using CoGroupByKey the workers run out of memory.
I have tried to do that using batches or use a local fixed window but
nothing is working well for me.
Can someone suggest what's the better way to merge such huge pcollections
with running OOM.

I appreciate your help.

Re: Need Help

[David Cavazos <dc...@google.com>]

Have you tried Flatten?
https://beam.apache.org/documentation/programming-guide/#flatten

On Mon, Feb 14, 2022 at 8:02 PM Pramod Kumar <de...@gmail.com>
wrote:

> Hi there,
> Hope everyone is doing well.
> I am stuck at a problem where I have to merge two pcollections one of it
> is pretty huge.
> While merging both using CoGroupByKey the workers run out of memory.
> I have tried to do that using batches or use a local fixed window but
> nothing is working well for me.
> Can someone suggest what's the better way to merge such huge pcollections
> with running OOM.
>
> I appreciate your help.
>

Re: Need Help

[Luke Cwik <lc...@google.com>]

If the other PCollection is much smaller, have you considered using it as a
side input?
Also, just to make sure you do mean you want to do a join across the two
PCollections because they share a common key and not that you're trying to
flatten them.

On Mon, Feb 14, 2022 at 8:02 PM Pramod Kumar <de...@gmail.com>
wrote:

> Hi there,
> Hope everyone is doing well.
> I am stuck at a problem where I have to merge two pcollections one of it
> is pretty huge.
> While merging both using CoGroupByKey the workers run out of memory.
> I have tried to do that using batches or use a local fixed window but
> nothing is working well for me.
> Can someone suggest what's the better way to merge such huge pcollections
> with running OOM.
>
> I appreciate your help.
>