You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Tobias Feldhaus <To...@localsearch.ch> on 2017/02/21 11:03:33 UTC
How to use the new Coder interface / TableRowJsonCoder in 0.5.0?
Hi,
According to this commit [0] it seems to me that there were some changes in the way the Coder interface is
being used.
Before switching from the Dataflow SDK 1.9 this code was working for me (I already adjusted the way PubSubIO is accessed):
.apply("WriteToPubSub", PubsubIO.write()
.topic(topic)
.withCoder(TableRowJsonCoder.of())
.timestampLabel("ts"));
Now withCoder(Coder<T> coder) no longer accepts the instance of TableRowJsonCoder. I have checked that
the TableRowJsonCoder is still registered as a StandardCoder [1] in the current master - how is it being used now?
Thank you for all your effort,
Tobi
[0] https://github.com/apache/beam/commit/7b98fa08d14e8121e8885f00a9a9a878b73f81a6
[1] https://github.com/apache/beam/blob/d9657ffc37490b063835672e0b5287b4d18aba96/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L94
Re: How to use the new Coder interface / TableRowJsonCoder in 0.5.0?
Posted by Thomas Groh <tg...@google.com>.
Hey Tobi;
I believe if you provide the type argument to the call to write()
everything will typecheck properly (given that the input PCollection is a
PCollection<TableRow>), e.g.
PubsubIO.<TableRow>write().topic(topic).withCoder(TableRowJsonCoder.of())...
The value returned by PubsubIO.write() is a generic type, but Java (at
least some versions) doesn't have the ability to figure out the type the
generic should be unless it's immediately assigned, hence the requirement
you specify it as early as possible. The method calls after that merely
propagate the type variable, which is why the call to withCoder doesn't
accept a Coder<TableRow>.
On Tue, Feb 21, 2017 at 5:12 AM, Tobias Feldhaus <
Tobias.Feldhaus@localsearch.ch> wrote:
> Hi Jean-Baptiste,
>
> I am using the standard TableRowJsonCoder of the SDK version 0.5.0.
> According to the official
> Javadoc [0] it looks like the description for encode() and decode() has
> even been copied from the Coder interface itself,
> am I missing something?
>
> Thank you
> Tobi
>
> https://beam.apache.org/documentation/sdks/javadoc/0.
> 5.0/org/apache/beam/sdk/coders/TableRowJsonCoder.html
>
> On 21 Feb 2017, at 13:06, Jean-Baptiste Onofré <jb...@nanthrax.net> wrote:
>
> Hi Tobias,
>
> PubusbIO accepts .withCoder(Coder<T> coder) with T is the type of the
> PCollection elements (PCollection<T>).
>
> So, you TableRowJsonCoder should be a class implementing Coder interface
> with Type T. Is it the case ?
>
> Thanks,
> Regards
> JB
>
> On 02/21/2017 12:03 PM, Tobias Feldhaus wrote:
>
> Hi,
>
> According to this commit [0] it seems to me that there were some changes
> in the way the Coder interface is
> being used.
>
> Before switching from the Dataflow SDK 1.9 this code was working for me
> (I already adjusted the way PubSubIO is accessed):
>
> .apply("WriteToPubSub", PubsubIO.write()
> .topic(topic)
> .withCoder(TableRowJsonCoder.of())
> .timestampLabel("ts"));
>
> Now withCoder(Coder<T> coder) no longer accepts the instance of
> TableRowJsonCoder. I have checked that
> the TableRowJsonCoder is still registered as a StandardCoder [1] in the
> current master - how is it being used now?
>
> Thank you for all your effort,
>
> Tobi
>
>
> [0] https://github.com/apache/beam/commit/7b98fa08d14e8121e8885f00a9a9a8
> 78b73f81a6
> [1] https://github.com/apache/beam/blob/d9657ffc37490b063835672e0b5287
> b4d18aba96/sdks/java/core/src/main/java/org/apache/beam/sdk/
> coders/CoderRegistry.java#L94
>
>
> --
> Jean-Baptiste Onofré
> jbonofre@apache.org
> http://blog.nanthrax.net
> Talend - http://www.talend.com
>
>
>
Re: How to use the new Coder interface / TableRowJsonCoder in 0.5.0?
Posted by Tobias Feldhaus <To...@localsearch.ch>.
Hi Jean-Baptiste,
I am using the standard TableRowJsonCoder of the SDK version 0.5.0. According to the official
Javadoc [0] it looks like the description for encode() and decode() has even been copied from the Coder interface itself,
am I missing something?
Thank you
Tobi
https://beam.apache.org/documentation/sdks/javadoc/0.5.0/org/apache/beam/sdk/coders/TableRowJsonCoder.html
On 21 Feb 2017, at 13:06, Jean-Baptiste Onofré <jb...@nanthrax.net>> wrote:
Hi Tobias,
PubusbIO accepts .withCoder(Coder<T> coder) with T is the type of the PCollection elements (PCollection<T>).
So, you TableRowJsonCoder should be a class implementing Coder interface with Type T. Is it the case ?
Thanks,
Regards
JB
On 02/21/2017 12:03 PM, Tobias Feldhaus wrote:
Hi,
According to this commit [0] it seems to me that there were some changes
in the way the Coder interface is
being used.
Before switching from the Dataflow SDK 1.9 this code was working for me
(I already adjusted the way PubSubIO is accessed):
.apply("WriteToPubSub", PubsubIO.write()
.topic(topic)
.withCoder(TableRowJsonCoder.of())
.timestampLabel("ts"));
Now withCoder(Coder<T> coder) no longer accepts the instance of
TableRowJsonCoder. I have checked that
the TableRowJsonCoder is still registered as a StandardCoder [1] in the
current master - how is it being used now?
Thank you for all your effort,
Tobi
[0] https://github.com/apache/beam/commit/7b98fa08d14e8121e8885f00a9a9a878b73f81a6
[1] https://github.com/apache/beam/blob/d9657ffc37490b063835672e0b5287b4d18aba96/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L94
--
Jean-Baptiste Onofré
jbonofre@apache.org<ma...@apache.org>
http://blog.nanthrax.net
Talend - http://www.talend.com
Re: How to use the new Coder interface / TableRowJsonCoder in 0.5.0?
Posted by Jean-Baptiste Onofré <jb...@nanthrax.net>.
Hi Tobias,
PubusbIO accepts .withCoder(Coder<T> coder) with T is the type of the
PCollection elements (PCollection<T>).
So, you TableRowJsonCoder should be a class implementing Coder interface
with Type T. Is it the case ?
Thanks,
Regards
JB
On 02/21/2017 12:03 PM, Tobias Feldhaus wrote:
> Hi,
>
> According to this commit [0] it seems to me that there were some changes
> in the way the Coder interface is
> being used.
>
> Before switching from the Dataflow SDK 1.9 this code was working for me
> (I already adjusted the way PubSubIO is accessed):
>
> .apply("WriteToPubSub", PubsubIO.write()
> .topic(topic)
> .withCoder(TableRowJsonCoder.of())
> .timestampLabel("ts"));
>
> Now withCoder(Coder<T> coder) no longer accepts the instance of
> TableRowJsonCoder. I have checked that
> the TableRowJsonCoder is still registered as a StandardCoder [1] in the
> current master - how is it being used now?
>
> Thank you for all your effort,
>
> Tobi
>
>
> [0] https://github.com/apache/beam/commit/7b98fa08d14e8121e8885f00a9a9a878b73f81a6
> [1] https://github.com/apache/beam/blob/d9657ffc37490b063835672e0b5287b4d18aba96/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/CoderRegistry.java#L94
--
Jean-Baptiste Onofr
jbonofre@apache.org
http://blog.nanthrax.net
Talend - http://www.talend.com