You are viewing a plain text version of this content. The canonical link for it is here.

Posted to dev@beam.apache.org by 8 Gianfortoni <8...@tokentransit.com> on 2018/05/04 18:41:15 UTC

Reading all elements of a PCollection after running beam go pipeline

Hi dev team,

I would like to be able to read the entire results of a PCollection
serially after running beam. In other frameworks this is fairly
straightforward, but I don't understand how one might do this with the Beam
Go SDK.

I guess I can read in a file that I write, but I want to be able to read
the elements in struct format. I looked through a bunch of examples but
don't see any that do this.

Thanks,
8

Re: Reading all elements of a PCollection after running beam go pipeline

Posted by Henning Rohde <he...@google.com>.

Great!

On Fri, May 4, 2018 at 4:37 PM 8 Gianfortoni <8...@tokentransit.com> wrote:

> Thanks for the workaround! That should work for me.
>
> On Fri, May 4, 2018, 1:51 PM Henning Rohde <he...@google.com> wrote:
>
>> Hey there,
>>
>>  Until side input is fully supported, you can use GBK with a fixed key to
>> get all elements in a single bundle (assuming global windowing). That is
>> how textio.Write works internally to produce a single file currently:
>>
>> https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144
>>
>>
>>
>> Thanks,
>>  Henning
>>
>>
>> On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote:
>>
>>> Hi dev team,
>>>
>>> I would like to be able to read the entire results of a PCollection
>>> serially after running beam. In other frameworks this is fairly
>>> straightforward, but I don't understand how one might do this with the Beam
>>> Go SDK.
>>>
>>> I guess I can read in a file that I write, but I want to be able to read
>>> the elements in struct format. I looked through a bunch of examples but
>>> don't see any that do this.
>>>
>>> Thanks,
>>> 8
>>>
>>

Re: Reading all elements of a PCollection after running beam go pipeline

Posted by 8 Gianfortoni <8...@tokentransit.com>.

Thanks for the workaround! That should work for me.

On Fri, May 4, 2018, 1:51 PM Henning Rohde <he...@google.com> wrote:

> Hey there,
>
>  Until side input is fully supported, you can use GBK with a fixed key to
> get all elements in a single bundle (assuming global windowing). That is
> how textio.Write works internally to produce a single file currently:
>
> https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144
>
>
>
> Thanks,
>  Henning
>
>
> On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote:
>
>> Hi dev team,
>>
>> I would like to be able to read the entire results of a PCollection
>> serially after running beam. In other frameworks this is fairly
>> straightforward, but I don't understand how one might do this with the Beam
>> Go SDK.
>>
>> I guess I can read in a file that I write, but I want to be able to read
>> the elements in struct format. I looked through a bunch of examples but
>> don't see any that do this.
>>
>> Thanks,
>> 8
>>
>

Re: Reading all elements of a PCollection after running beam go pipeline

Posted by Henning Rohde <he...@google.com>.

Hey there,

 Until side input is fully supported, you can use GBK with a fixed key to
get all elements in a single bundle (assuming global windowing). That is
how textio.Write works internally to produce a single file currently:

https://github.com/apache/beam/blob/90ba47114997e75a3a7daa1e8db92768d05e5432/sdks/go/pkg/beam/io/textio/textio.go#L144



Thanks,
 Henning


On Fri, May 4, 2018 at 11:41 AM 8 Gianfortoni <8...@tokentransit.com> wrote:

> Hi dev team,
>
> I would like to be able to read the entire results of a PCollection
> serially after running beam. In other frameworks this is fairly
> straightforward, but I don't understand how one might do this with the Beam
> Go SDK.
>
> I guess I can read in a file that I write, but I want to be able to read
> the elements in struct format. I looked through a bunch of examples but
> don't see any that do this.
>
> Thanks,
> 8
>