ptransform of two views and no data

[Antony Mayi <an...@yahoo.com>]

Hi,
what is the best way to do some operation combining two PCollectionView instances that's not involving any PCollection processing? So far I've been running it on some fake PCollection of single element and passing the views as sideinputs:
pipeline.apply(Create.of(fake_element)).apply(ParDo.of(...).withSideInputs(view1, view2)))...
but the need for fake pcollection is bit awkward so wondering if it can be avoided?
I can see in beam v2.0.0 there is PCollectionView.getPCollection() so I could theoretically use one of the views using main input and the other as sideinput... but the .getPCollection() is annotated as internal so I guess that's not to be used.
thanks,antony.

Re: ptransform of two views and no data

[Dan Halperin <dh...@google.com>]

Hi Antony,

Can you say a little more about your use case? There are many approaches to
making this work – your solution is actually a valid and often-used
technique – but yes you might use one collection as the main input (just
don't create a view), or you might use a CoGroupByKey and no side inputs,
or maybe other things.

Thanks,
Dan

On Wed, May 17, 2017 at 5:24 AM, Antony Mayi <an...@yahoo.com> wrote:

> Hi,
>
> what is the best way to do some operation combining two PCollectionView
> instances that's not involving any PCollection processing? So far I've been
> running it on some fake PCollection of single element and passing the views
> as sideinputs:
>
> pipeline.apply(Create.of(fake_element)).apply(ParDo.of(...).withSideInputs(view1,
> view2)))...
>
> but the need for fake pcollection is bit awkward so wondering if it can be
> avoided?
>
> I can see in beam v2.0.0 there is PCollectionView.getPCollection() so I
> could theoretically use one of the views using main input and the other as
> sideinput... but the .getPCollection() is annotated as internal so I guess
> that's not to be used.
>
> thanks,
> antony.
>