You are viewing a plain text version of this content. The canonical link for it is here.

Posted to user@beam.apache.org by kaniska Mandal <ka...@gmail.com> on 2016/05/13 22:00:17 UTC

Beam-Flink-Runner example using Protocol Buffer

Hi Max,

It would be great if any example can be provided - how we can encode and
decode streaming messages using Google Protocol Buffer in a Pipeline using
Flink-Runner.


Thanks,
Kaniska

Re: Beam-Flink-Runner example using Protocol Buffer

Posted by Maximilian Michels <mx...@apache.org>.

Hi Kaniska, hi Frances,

The Flink Runner uses the Beam type information alongside with its
coders. So using the ProtoCoder should work fine. If you're using
Kafka for receiving/sending records, you can proceed similarly as in
the AvroKafka example, replacing the AvroCoder with the ProtoCoder.

Cheers,
Max

On Sat, May 14, 2016 at 12:33 AM, Frances Perry <fj...@google.com> wrote:
> The Beam CoderRegistry registers ProtoCoder for encoding
> PCollection<Message>. So the goal would be that the type inference should
> correctly figure out how to encode things at the Beam level and then work in
> all runners. (But I don't know the details on the FlinkRunner.)
>
> On Fri, May 13, 2016 at 3:00 PM, kaniska Mandal <ka...@gmail.com>
> wrote:
>>
>> Hi Max,
>>
>> It would be great if any example can be provided - how we can encode and
>> decode streaming messages using Google Protocol Buffer in a Pipeline using
>> Flink-Runner.
>>
>>
>> Thanks,
>> Kaniska
>
>

Re: Beam-Flink-Runner example using Protocol Buffer

Posted by Frances Perry <fj...@google.com>.

The Beam CoderRegistry registers ProtoCoder
<https://github.com/apache/incubator-beam/blob/96765f19b1bd8149240cd77eb7cf7fb636e477e4/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/protobuf/ProtoCoder.java>
for encoding PCollection<Message>. So the goal would be that the type
inference should correctly figure out how to encode things at the Beam
level and then work in all runners. (But I don't know the details on the
FlinkRunner.)

On Fri, May 13, 2016 at 3:00 PM, kaniska Mandal <ka...@gmail.com>
wrote:

> Hi Max,
>
> It would be great if any example can be provided - how we can encode and
> decode streaming messages using Google Protocol Buffer in a Pipeline using
> Flink-Runner.
>
>
> Thanks,
> Kaniska
>