Utility for Serializing PipelineOptions

["P. Taylor Goetz" <pt...@apache.org>]

Looking at the runner implementations for Spark, Flink, and Gearpump, they all have the same requirement of being able to serialize/deserialize PipelineOptions, and they each seem to use slightly different approaches. In working on a Beam runner for Storm I have the same requirement, and imagine other runner implementations might as well.

Would it make sense to add this functionality in the core SDK?

-Taylor

Re: Utility for Serializing PipelineOptions

[Manu Zhang <ow...@gmail.com>]

Hi Lukasz,

For both Gearpump DAG and Storm Topology, they are built and serialized at
client side, and sent to server.  For things (e.g. Reader) that depends on
PipelineOptions to create at server side, they require PipelineOptions to
be serialized at client side. (example code
<https://github.com/apache/incubator-beam/blob/gearpump-runner/runners/gearpump/src/main/java/org/apache/beam/runners/gearpump/translators/io/GearpumpSource.java#L50>
)

Is this what you mean, Taylor ? I'm not sure core Java SDK can do this part
for runners since not all runners adopt the serde way for application
submit.


On Thu, Aug 4, 2016 at 4:45 AM Lukasz Cwik <lc...@google.com.invalid> wrote:

> The core Java SDK already contains the code to serialize/deserialize
> PipelineOptions using Jackson's ObjectMapper since PipelineOptions has
> Jackson databind annotations @JsonSerialize and @JsonDeserialize.
>
> Is there something more specific that you were looking for?
>
> On Wed, Aug 3, 2016 at 12:54 PM, P. Taylor Goetz <pt...@apache.org>
> wrote:
>
> > Looking at the runner implementations for Spark, Flink, and Gearpump,
> they
> > all have the same requirement of being able to serialize/deserialize
> > PipelineOptions, and they each seem to use slightly different approaches.
> > In working on a Beam runner for Storm I have the same requirement, and
> > imagine other runner implementations might as well.
> >
> > Would it make sense to add this functionality in the core SDK?
> >
> > -Taylor
> >
>

Re: Utility for Serializing PipelineOptions

[Lukasz Cwik <lc...@google.com.INVALID>]

The core Java SDK already contains the code to serialize/deserialize
PipelineOptions using Jackson's ObjectMapper since PipelineOptions has
Jackson databind annotations @JsonSerialize and @JsonDeserialize.

Is there something more specific that you were looking for?

On Wed, Aug 3, 2016 at 12:54 PM, P. Taylor Goetz <pt...@apache.org> wrote:

> Looking at the runner implementations for Spark, Flink, and Gearpump, they
> all have the same requirement of being able to serialize/deserialize
> PipelineOptions, and they each seem to use slightly different approaches.
> In working on a Beam runner for Storm I have the same requirement, and
> imagine other runner implementations might as well.
>
> Would it make sense to add this functionality in the core SDK?
>
> -Taylor
>