You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Matthias Baetens <ba...@gmail.com> on 2019/01/31 09:14:36 UTC
Beam Python streaming pipeline on Flink Runner
Hi everyone,
Last few days I have been trying to run a streaming pipeline (code on Github
<https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
I am running a Flink cluster locally (v1.5.6
<https://flink.apache.org/downloads.html>)
I have built the SDK Harness Container: *./gradlew
:beam-sdks-python-container:docker*
and started the JobServer: *./gradlew
:beam-runners-flink_2.11-job-server:runShadow
-PflinkMasterUrl=localhost:8081.*
I run my pipeline with: *env/bin/python streaming_pipeline.py
--runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
--input_subscription xxx --output_subscription xxx*
All this is running inside a Ubuntu (Bionic) in a Virtualbox.
The job submits fine, but unfortunately fails after a few seconds with the
error attached.
Anything I am missing or doing wrong?
Many thanks.
Best,
Matthias
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
And, as Max says, this is an SDF that wraps a BoundedSource or
UnboundedSource respectively. The other way around is not possible, as
SDF is strictly more powerful.
On Thu, Jan 31, 2019 at 3:52 PM Robert Bradshaw <ro...@google.com> wrote:
>
> On Thu, Jan 31, 2019 at 3:17 PM Ismaël Mejía <ie...@gmail.com> wrote:
> >
> > > Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration.
> >
> > That would be fantastic, I have heard about such wrapper multiple
> > times but so far there is not any realistic proposal. I have a hard
> > time to imagine how can we map in a generic way RestrictionTrackers
> > into the existing Bounded/UnboundedSource, so I would love to hear
> > more about the details.
>
> For BoundedSource the restriction is the BoundedSource object itself
> (which splits into multiple distinct bounded sources) with a tracker
> that forwards split calls to the reader, and the body of process would
> read from this reader to completion (never explicitly claiming
> positions from the reader).
>
> For unbounded sources, the restriction tracker always returns true on
> try_claim, false on try_split, and the process method returns current
> elements until either advance or try_claim returns false or try_split
> was called at 0, and the checkpoint mark is returned as the checkpoint
> residual.
>
> Initial splitting and sizing just forwards the calls.
>
> > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org> wrote:
> > >
> > > > In addition to have support in the runners, this will require a
> > > > rewrite of PubsubIO to use the new SDF API.
> > >
> > > Not necessarily. This would be one way. Another way is build an SDF wrapper for
> > > UnboundedSource. Probably the easier path for migration.
> > >
> > > On 31.01.19 14:03, Ismaël Mejía wrote:
> > > >> Fortunately, there is already a pending PR for cross-language pipelines which
> > > >> will allow us to use Java IO like PubSub in Python jobs.
> > > >
> > > > In addition to have support in the runners, this will require a
> > > > rewrite of PubsubIO to use the new SDF API.
> > > >
> > > > On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
> > > >>
> > > >> Hi Matthias,
> > > >>
> > > >> This is already reflected in the compatibility matrix, if you look under SDF.
> > > >> There is no UnboundedSource interface for portable pipelines. That's a legacy
> > > >> abstraction that will be replaced with SDF.
> > > >>
> > > >> Fortunately, there is already a pending PR for cross-language pipelines which
> > > >> will allow us to use Java IO like PubSub in Python jobs.
> > > >>
> > > >> Thanks,
> > > >> Max
> > > >>
> > > >> On 31.01.19 12:06, Matthias Baetens wrote:
> > > >>> Hey Ankur,
> > > >>>
> > > >>> Thanks for the swift reply. Should I change this in the capability matrix
> > > >>> <https://s.apache.org/apache-beam-portability-support-table> then?
> > > >>>
> > > >>> Many thanks.
> > > >>> Best,
> > > >>> Matthias
> > > >>>
> > > >>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> > > >>> <ma...@google.com>> wrote:
> > > >>>
> > > >>> Hi Matthias,
> > > >>>
> > > >>> Unfortunately, unbounded reads including pubsub are not yet supported for
> > > >>> portable runners.
> > > >>>
> > > >>> Thanks,
> > > >>> Ankur
> > > >>>
> > > >>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> > > >>> <ma...@gmail.com>> wrote:
> > > >>>
> > > >>> Hi everyone,
> > > >>>
> > > >>> Last few days I have been trying to run a streaming pipeline (code on
> > > >>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
> > > >>>
> > > >>> I am running a Flink cluster locally (v1.5.6
> > > >>> <https://flink.apache.org/downloads.html>)
> > > >>> I have built the SDK Harness Container: /./gradlew
> > > >>> :beam-sdks-python-container:docker/
> > > >>> and started the JobServer: /./gradlew
> > > >>> :beam-runners-flink_2.11-job-server:runShadow
> > > >>> -PflinkMasterUrl=localhost:8081./
> > > >>>
> > > >>> I run my pipeline with: /env/bin/python streaming_pipeline.py
> > > >>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> > > >>> --input_subscription xxx --output_subscription xxx/
> > > >>> /
> > > >>> /
> > > >>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
> > > >>>
> > > >>> The job submits fine, but unfortunately fails after a few seconds with
> > > >>> the error attached.
> > > >>>
> > > >>> Anything I am missing or doing wrong?
> > > >>>
> > > >>> Many thanks.
> > > >>> Best,
> > > >>> Matthias
> > > >>>
> > > >>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Jan 31, 2019 at 3:17 PM Ismaël Mejía <ie...@gmail.com> wrote:
>
> > Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration.
>
> That would be fantastic, I have heard about such wrapper multiple
> times but so far there is not any realistic proposal. I have a hard
> time to imagine how can we map in a generic way RestrictionTrackers
> into the existing Bounded/UnboundedSource, so I would love to hear
> more about the details.
For BoundedSource the restriction is the BoundedSource object itself
(which splits into multiple distinct bounded sources) with a tracker
that forwards split calls to the reader, and the body of process would
read from this reader to completion (never explicitly claiming
positions from the reader).
For unbounded sources, the restriction tracker always returns true on
try_claim, false on try_split, and the process method returns current
elements until either advance or try_claim returns false or try_split
was called at 0, and the checkpoint mark is returned as the checkpoint
residual.
Initial splitting and sizing just forwards the calls.
> On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org> wrote:
> >
> > > In addition to have support in the runners, this will require a
> > > rewrite of PubsubIO to use the new SDF API.
> >
> > Not necessarily. This would be one way. Another way is build an SDF wrapper for
> > UnboundedSource. Probably the easier path for migration.
> >
> > On 31.01.19 14:03, Ismaël Mejía wrote:
> > >> Fortunately, there is already a pending PR for cross-language pipelines which
> > >> will allow us to use Java IO like PubSub in Python jobs.
> > >
> > > In addition to have support in the runners, this will require a
> > > rewrite of PubsubIO to use the new SDF API.
> > >
> > > On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
> > >>
> > >> Hi Matthias,
> > >>
> > >> This is already reflected in the compatibility matrix, if you look under SDF.
> > >> There is no UnboundedSource interface for portable pipelines. That's a legacy
> > >> abstraction that will be replaced with SDF.
> > >>
> > >> Fortunately, there is already a pending PR for cross-language pipelines which
> > >> will allow us to use Java IO like PubSub in Python jobs.
> > >>
> > >> Thanks,
> > >> Max
> > >>
> > >> On 31.01.19 12:06, Matthias Baetens wrote:
> > >>> Hey Ankur,
> > >>>
> > >>> Thanks for the swift reply. Should I change this in the capability matrix
> > >>> <https://s.apache.org/apache-beam-portability-support-table> then?
> > >>>
> > >>> Many thanks.
> > >>> Best,
> > >>> Matthias
> > >>>
> > >>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> > >>> <ma...@google.com>> wrote:
> > >>>
> > >>> Hi Matthias,
> > >>>
> > >>> Unfortunately, unbounded reads including pubsub are not yet supported for
> > >>> portable runners.
> > >>>
> > >>> Thanks,
> > >>> Ankur
> > >>>
> > >>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> > >>> <ma...@gmail.com>> wrote:
> > >>>
> > >>> Hi everyone,
> > >>>
> > >>> Last few days I have been trying to run a streaming pipeline (code on
> > >>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
> > >>>
> > >>> I am running a Flink cluster locally (v1.5.6
> > >>> <https://flink.apache.org/downloads.html>)
> > >>> I have built the SDK Harness Container: /./gradlew
> > >>> :beam-sdks-python-container:docker/
> > >>> and started the JobServer: /./gradlew
> > >>> :beam-runners-flink_2.11-job-server:runShadow
> > >>> -PflinkMasterUrl=localhost:8081./
> > >>>
> > >>> I run my pipeline with: /env/bin/python streaming_pipeline.py
> > >>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> > >>> --input_subscription xxx --output_subscription xxx/
> > >>> /
> > >>> /
> > >>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
> > >>>
> > >>> The job submits fine, but unfortunately fails after a few seconds with
> > >>> the error attached.
> > >>>
> > >>> Anything I am missing or doing wrong?
> > >>>
> > >>> Many thanks.
> > >>> Best,
> > >>> Matthias
> > >>>
> > >>>
Re: Cross-language transform API
Posted by Reuven Lax <re...@google.com>.
About schemas, pr/7952 adds schemas to the runner API.
This PR focuses on the Dataflow runner (not the portability runner).
However after some community discussion on the design (we are still free to
change these protos based on discussion), I would like to stabilize these
protos.
On Mon, Mar 11, 2019 at 11:04 AM Maximilian Michels <mx...@apache.org> wrote:
> Thanks for the remarks. Correct, we do not need the static URN at all in
> the payload. We can pass the transform URN with the ExternalTransform as
> part of the ExpansionRequest. So this is sufficient for the Proto:
>
> message ConfigValue {
> string coder_urn = 1;
> bytes payload = 2;
> }
>
> message ExternalTransformPayload {
> map<string, ConfigValue> configuration = 1;
> }
>
>
> Considering Schemas, I'm not sure they are useful for the scope of the
> PR. I think basic Java Reflection is enough.
>
> Thanks,
> Max
>
> On 11.03.19 18:36, Robert Bradshaw wrote:
> > On Mon, Mar 11, 2019 at 6:05 PM Chamikara Jayalath <ch...@google.com>
> wrote:
> >>
> >> On Mon, Mar 11, 2019 at 9:27 AM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>
> >>> On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>>>
> >>>>> Just to clarify. What's the reason for including a PROPERTIES enum
> here instead of directly making beam_urn a field of
> ExternalTransformPayload ?
> >>>>
> >>>> The URN is supposed to be static. We always use the same URN for this
> >>>> type of external transform. We probably want an additional identifier
> to
> >>>> point to the resource we want to configure.
> >>>
> >>> It does feel odd to not use the URN to specify the transform itself,
> >>> and embed the true identity in an inner proto. The notion of
> >>> "external" is just how it happens to be invoked in this pipeline, not
> >>> part of its intrinsic definition. As we want introspection
> >>> capabilities in the service, we should be able to use the URN at a top
> >>> level and know what kind of payload it expects. I would also like to
> >>> see this kind of information populated for non-extern transforms which
> >>> could be good for visibility (substitution, visualization, etc.) for
> >>> runners and other pipeline-consuming tools.
> >>>
> >>>> Like so:
> >>>>
> >>>> message ExternalTransformPayload {
> >>>> enum Enum {
> >>>> PROPERTIES = 0
> >>>> [(beam_urn) =
> "beam:external:transform:external_transform:v1"];
> >>>> }
> >>>> // A fully-qualified identifier, e.g. Java package + class
> >>>> string identifier = 1;
> >>>
> >>> I'd rather the identifier have semantic rather than
> >>> implementation-specific meaning. e.g. one could imagine multiple
> >>> implementations of a given transform that different services could
> >>> offer.
> >>>
> >>>> // the format may change to map<string, bytes> if types are
> supported
> >>>> map<string, string> parameters = 2;
> >>>> }
> >>>>
> >>>> The identifier could also be a URN.
> >>>>
> >>>>> Can we change first version to map<string, bytes> ? Otherwise the
> set of transforms we can support/test will be very limited.
> >>>>
> >>>> How do we do that? Do we define a set of standard coders for supported
> >>>> types? On the Java side we can lookup the coder by extracting the
> field
> >>>> from the Pojo, but we can't do that in Python.
> >>
> >>
> >> I'll let Reuven comment on exact relevance and timelines on Beam Schema
> related work here but till we have that probably we can support the
> standard set of coders that are well defined here ?
> >>
> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
> >>
> >> So in Python side the ExternalTransform can take a list of parameters
> (of types that have standard coders) which will be converted to bytes to be
> sent over the wire. In Java side corresponding standard coders (which are
> determined by introspection of transform builder's payload POJO) can be
> used to covert bytes to objects.
> >
> > They also need to agree on the field types as well as names, so would
> > it be map<name, pair<coder_id, bytes>>. I'm not sure the tradeoff
> > between going further down this road vs. getting schemas up to par in
> > Python (and, next, Go). And supporting this long term in parallel to
> > what we come up with schemas.
> >
> >> Hopefully Beam schema work will give us a more generalized way to
> convert objects across languages (for example, Python object -> Python Row
> + Schema -> Java Row + Schema -> Java object). Note that we run into the
> same issue when data tries to cross SDK boundaries when executing
> cross-language pipelines.
> >
> > +1, which is another reason I want to accelerate the language
> > independence of schemas.
> >
> >>>>> Can we re-use some of the Beam schemas-related work/utilities here ?
> >>>>
> >>>> Yes, that was the plan.
> >>>
> >>> On this note, Reuven, what is the plan (and timeline) for a
> >>> language-independent representation of schemas? The crux of the
> >>> problem is that the user needs to specify some kind of configuration
> >>> (call it C) to construct the transform (call it T). This would be
> >>> handled by a TransformBuilder<C, T> that provides (at least) a mapping
> >>> C -> T. (Possibly this interface could be offered on the transform
> >>> itself).
> >>>
> >>> The question we are trying to answer here is how to represent C, in
> >>> both the source and target language, and on the wire. The idea is that
> >>> we could leverage the schema infrastructure such that C could be a
> >>> POJO in Java (and perhaps a dict in Python). We would want to extend
> >>> Schemas and Row (or perhaps a sub/super/sibling class thereof) to
> >>> allow for Coder and UDF-typed fields. (Exactly how to represent UDFs
> >>> is still very TBD.) The payload for a external transform using this
> >>> format would be the tuple (schema, SchemaCoder(schema).encode(C)). The
> >>> goal is to not, yet again, invent a cross-language way of defining a
> >>> bag of named, typed parameters (aka fields) with language-idiomatic
> >>> mappings and some introspection capabilities, and significantly less
> >>> heavy-weight than users defining their own protos (plus generating
> >>> bindings to all languages).
> >>>
> >>> Does this seem a reasonable use of schemas?
>
Re: Cross-language transform API
Posted by Maximilian Michels <mx...@apache.org>.
Thanks for the remarks. Correct, we do not need the static URN at all in
the payload. We can pass the transform URN with the ExternalTransform as
part of the ExpansionRequest. So this is sufficient for the Proto:
message ConfigValue {
string coder_urn = 1;
bytes payload = 2;
}
message ExternalTransformPayload {
map<string, ConfigValue> configuration = 1;
}
Considering Schemas, I'm not sure they are useful for the scope of the
PR. I think basic Java Reflection is enough.
Thanks,
Max
On 11.03.19 18:36, Robert Bradshaw wrote:
> On Mon, Mar 11, 2019 at 6:05 PM Chamikara Jayalath <ch...@google.com> wrote:
>>
>> On Mon, Mar 11, 2019 at 9:27 AM Robert Bradshaw <ro...@google.com> wrote:
>>>
>>> On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels <mx...@apache.org> wrote:
>>>>
>>>>> Just to clarify. What's the reason for including a PROPERTIES enum here instead of directly making beam_urn a field of ExternalTransformPayload ?
>>>>
>>>> The URN is supposed to be static. We always use the same URN for this
>>>> type of external transform. We probably want an additional identifier to
>>>> point to the resource we want to configure.
>>>
>>> It does feel odd to not use the URN to specify the transform itself,
>>> and embed the true identity in an inner proto. The notion of
>>> "external" is just how it happens to be invoked in this pipeline, not
>>> part of its intrinsic definition. As we want introspection
>>> capabilities in the service, we should be able to use the URN at a top
>>> level and know what kind of payload it expects. I would also like to
>>> see this kind of information populated for non-extern transforms which
>>> could be good for visibility (substitution, visualization, etc.) for
>>> runners and other pipeline-consuming tools.
>>>
>>>> Like so:
>>>>
>>>> message ExternalTransformPayload {
>>>> enum Enum {
>>>> PROPERTIES = 0
>>>> [(beam_urn) = "beam:external:transform:external_transform:v1"];
>>>> }
>>>> // A fully-qualified identifier, e.g. Java package + class
>>>> string identifier = 1;
>>>
>>> I'd rather the identifier have semantic rather than
>>> implementation-specific meaning. e.g. one could imagine multiple
>>> implementations of a given transform that different services could
>>> offer.
>>>
>>>> // the format may change to map<string, bytes> if types are supported
>>>> map<string, string> parameters = 2;
>>>> }
>>>>
>>>> The identifier could also be a URN.
>>>>
>>>>> Can we change first version to map<string, bytes> ? Otherwise the set of transforms we can support/test will be very limited.
>>>>
>>>> How do we do that? Do we define a set of standard coders for supported
>>>> types? On the Java side we can lookup the coder by extracting the field
>>>> from the Pojo, but we can't do that in Python.
>>
>>
>> I'll let Reuven comment on exact relevance and timelines on Beam Schema related work here but till we have that probably we can support the standard set of coders that are well defined here ?
>> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
>>
>> So in Python side the ExternalTransform can take a list of parameters (of types that have standard coders) which will be converted to bytes to be sent over the wire. In Java side corresponding standard coders (which are determined by introspection of transform builder's payload POJO) can be used to covert bytes to objects.
>
> They also need to agree on the field types as well as names, so would
> it be map<name, pair<coder_id, bytes>>. I'm not sure the tradeoff
> between going further down this road vs. getting schemas up to par in
> Python (and, next, Go). And supporting this long term in parallel to
> what we come up with schemas.
>
>> Hopefully Beam schema work will give us a more generalized way to convert objects across languages (for example, Python object -> Python Row + Schema -> Java Row + Schema -> Java object). Note that we run into the same issue when data tries to cross SDK boundaries when executing cross-language pipelines.
>
> +1, which is another reason I want to accelerate the language
> independence of schemas.
>
>>>>> Can we re-use some of the Beam schemas-related work/utilities here ?
>>>>
>>>> Yes, that was the plan.
>>>
>>> On this note, Reuven, what is the plan (and timeline) for a
>>> language-independent representation of schemas? The crux of the
>>> problem is that the user needs to specify some kind of configuration
>>> (call it C) to construct the transform (call it T). This would be
>>> handled by a TransformBuilder<C, T> that provides (at least) a mapping
>>> C -> T. (Possibly this interface could be offered on the transform
>>> itself).
>>>
>>> The question we are trying to answer here is how to represent C, in
>>> both the source and target language, and on the wire. The idea is that
>>> we could leverage the schema infrastructure such that C could be a
>>> POJO in Java (and perhaps a dict in Python). We would want to extend
>>> Schemas and Row (or perhaps a sub/super/sibling class thereof) to
>>> allow for Coder and UDF-typed fields. (Exactly how to represent UDFs
>>> is still very TBD.) The payload for a external transform using this
>>> format would be the tuple (schema, SchemaCoder(schema).encode(C)). The
>>> goal is to not, yet again, invent a cross-language way of defining a
>>> bag of named, typed parameters (aka fields) with language-idiomatic
>>> mappings and some introspection capabilities, and significantly less
>>> heavy-weight than users defining their own protos (plus generating
>>> bindings to all languages).
>>>
>>> Does this seem a reasonable use of schemas?
Re: Cross-language transform API
Posted by Robert Bradshaw <ro...@google.com>.
On Mon, Mar 11, 2019 at 6:05 PM Chamikara Jayalath <ch...@google.com> wrote:
>
> On Mon, Mar 11, 2019 at 9:27 AM Robert Bradshaw <ro...@google.com> wrote:
>>
>> On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels <mx...@apache.org> wrote:
>> >
>> > > Just to clarify. What's the reason for including a PROPERTIES enum here instead of directly making beam_urn a field of ExternalTransformPayload ?
>> >
>> > The URN is supposed to be static. We always use the same URN for this
>> > type of external transform. We probably want an additional identifier to
>> > point to the resource we want to configure.
>>
>> It does feel odd to not use the URN to specify the transform itself,
>> and embed the true identity in an inner proto. The notion of
>> "external" is just how it happens to be invoked in this pipeline, not
>> part of its intrinsic definition. As we want introspection
>> capabilities in the service, we should be able to use the URN at a top
>> level and know what kind of payload it expects. I would also like to
>> see this kind of information populated for non-extern transforms which
>> could be good for visibility (substitution, visualization, etc.) for
>> runners and other pipeline-consuming tools.
>>
>> > Like so:
>> >
>> > message ExternalTransformPayload {
>> > enum Enum {
>> > PROPERTIES = 0
>> > [(beam_urn) = "beam:external:transform:external_transform:v1"];
>> > }
>> > // A fully-qualified identifier, e.g. Java package + class
>> > string identifier = 1;
>>
>> I'd rather the identifier have semantic rather than
>> implementation-specific meaning. e.g. one could imagine multiple
>> implementations of a given transform that different services could
>> offer.
>>
>> > // the format may change to map<string, bytes> if types are supported
>> > map<string, string> parameters = 2;
>> > }
>> >
>> > The identifier could also be a URN.
>> >
>> > > Can we change first version to map<string, bytes> ? Otherwise the set of transforms we can support/test will be very limited.
>> >
>> > How do we do that? Do we define a set of standard coders for supported
>> > types? On the Java side we can lookup the coder by extracting the field
>> > from the Pojo, but we can't do that in Python.
>
>
> I'll let Reuven comment on exact relevance and timelines on Beam Schema related work here but till we have that probably we can support the standard set of coders that are well defined here ?
> https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
>
> So in Python side the ExternalTransform can take a list of parameters (of types that have standard coders) which will be converted to bytes to be sent over the wire. In Java side corresponding standard coders (which are determined by introspection of transform builder's payload POJO) can be used to covert bytes to objects.
They also need to agree on the field types as well as names, so would
it be map<name, pair<coder_id, bytes>>. I'm not sure the tradeoff
between going further down this road vs. getting schemas up to par in
Python (and, next, Go). And supporting this long term in parallel to
what we come up with schemas.
> Hopefully Beam schema work will give us a more generalized way to convert objects across languages (for example, Python object -> Python Row + Schema -> Java Row + Schema -> Java object). Note that we run into the same issue when data tries to cross SDK boundaries when executing cross-language pipelines.
+1, which is another reason I want to accelerate the language
independence of schemas.
>> > > Can we re-use some of the Beam schemas-related work/utilities here ?
>> >
>> > Yes, that was the plan.
>>
>> On this note, Reuven, what is the plan (and timeline) for a
>> language-independent representation of schemas? The crux of the
>> problem is that the user needs to specify some kind of configuration
>> (call it C) to construct the transform (call it T). This would be
>> handled by a TransformBuilder<C, T> that provides (at least) a mapping
>> C -> T. (Possibly this interface could be offered on the transform
>> itself).
>>
>> The question we are trying to answer here is how to represent C, in
>> both the source and target language, and on the wire. The idea is that
>> we could leverage the schema infrastructure such that C could be a
>> POJO in Java (and perhaps a dict in Python). We would want to extend
>> Schemas and Row (or perhaps a sub/super/sibling class thereof) to
>> allow for Coder and UDF-typed fields. (Exactly how to represent UDFs
>> is still very TBD.) The payload for a external transform using this
>> format would be the tuple (schema, SchemaCoder(schema).encode(C)). The
>> goal is to not, yet again, invent a cross-language way of defining a
>> bag of named, typed parameters (aka fields) with language-idiomatic
>> mappings and some introspection capabilities, and significantly less
>> heavy-weight than users defining their own protos (plus generating
>> bindings to all languages).
>>
>> Does this seem a reasonable use of schemas?
Re: Cross-language transform API
Posted by Chamikara Jayalath <ch...@google.com>.
On Mon, Mar 11, 2019 at 9:27 AM Robert Bradshaw <ro...@google.com> wrote:
> On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels <mx...@apache.org> wrote:
> >
> > > Just to clarify. What's the reason for including a PROPERTIES enum
> here instead of directly making beam_urn a field of
> ExternalTransformPayload ?
> >
> > The URN is supposed to be static. We always use the same URN for this
> > type of external transform. We probably want an additional identifier to
> > point to the resource we want to configure.
>
> It does feel odd to not use the URN to specify the transform itself,
> and embed the true identity in an inner proto. The notion of
> "external" is just how it happens to be invoked in this pipeline, not
> part of its intrinsic definition. As we want introspection
> capabilities in the service, we should be able to use the URN at a top
> level and know what kind of payload it expects. I would also like to
> see this kind of information populated for non-extern transforms which
> could be good for visibility (substitution, visualization, etc.) for
> runners and other pipeline-consuming tools.
>
> > Like so:
> >
> > message ExternalTransformPayload {
> > enum Enum {
> > PROPERTIES = 0
> > [(beam_urn) = "beam:external:transform:external_transform:v1"];
> > }
> > // A fully-qualified identifier, e.g. Java package + class
> > string identifier = 1;
>
> I'd rather the identifier have semantic rather than
> implementation-specific meaning. e.g. one could imagine multiple
> implementations of a given transform that different services could
> offer.
>
> > // the format may change to map<string, bytes> if types are supported
> > map<string, string> parameters = 2;
> > }
> >
> > The identifier could also be a URN.
> >
> > > Can we change first version to map<string, bytes> ? Otherwise the set
> of transforms we can support/test will be very limited.
> >
> > How do we do that? Do we define a set of standard coders for supported
> > types? On the Java side we can lookup the coder by extracting the field
> > from the Pojo, but we can't do that in Python.
>
I'll let Reuven comment on exact relevance and timelines on Beam Schema
related work here but till we have that probably we can support the
standard set of coders that are well defined here ?
https://github.com/apache/beam/blob/master/model/pipeline/src/main/proto/beam_runner_api.proto#L542
So in Python side the ExternalTransform can take a list of parameters (of
types that have standard coders) which will be converted to bytes to be
sent over the wire. In Java side corresponding standard coders (which are
determined by introspection of transform builder's payload POJO) can be
used to covert bytes to objects.
Hopefully Beam schema work will give us a more generalized way to convert
objects across languages (for example, Python object -> Python Row + Schema
-> Java Row + Schema -> Java object). Note that we run into the same issue
when data tries to cross SDK boundaries when executing cross-language
pipelines.
>
> > > Can we re-use some of the Beam schemas-related work/utilities here ?
> >
> > Yes, that was the plan.
>
> On this note, Reuven, what is the plan (and timeline) for a
> language-independent representation of schemas? The crux of the
> problem is that the user needs to specify some kind of configuration
> (call it C) to construct the transform (call it T). This would be
> handled by a TransformBuilder<C, T> that provides (at least) a mapping
> C -> T. (Possibly this interface could be offered on the transform
> itself).
>
> The question we are trying to answer here is how to represent C, in
> both the source and target language, and on the wire. The idea is that
> we could leverage the schema infrastructure such that C could be a
> POJO in Java (and perhaps a dict in Python). We would want to extend
> Schemas and Row (or perhaps a sub/super/sibling class thereof) to
> allow for Coder and UDF-typed fields. (Exactly how to represent UDFs
> is still very TBD.) The payload for a external transform using this
> format would be the tuple (schema, SchemaCoder(schema).encode(C)). The
> goal is to not, yet again, invent a cross-language way of defining a
> bag of named, typed parameters (aka fields) with language-idiomatic
> mappings and some introspection capabilities, and significantly less
> heavy-weight than users defining their own protos (plus generating
> bindings to all languages).
>
> Does this seem a reasonable use of schemas?
>
Re: Cross-language transform API
Posted by Robert Bradshaw <ro...@google.com>.
On Mon, Mar 11, 2019 at 4:37 PM Maximilian Michels <mx...@apache.org> wrote:
>
> > Just to clarify. What's the reason for including a PROPERTIES enum here instead of directly making beam_urn a field of ExternalTransformPayload ?
>
> The URN is supposed to be static. We always use the same URN for this
> type of external transform. We probably want an additional identifier to
> point to the resource we want to configure.
It does feel odd to not use the URN to specify the transform itself,
and embed the true identity in an inner proto. The notion of
"external" is just how it happens to be invoked in this pipeline, not
part of its intrinsic definition. As we want introspection
capabilities in the service, we should be able to use the URN at a top
level and know what kind of payload it expects. I would also like to
see this kind of information populated for non-extern transforms which
could be good for visibility (substitution, visualization, etc.) for
runners and other pipeline-consuming tools.
> Like so:
>
> message ExternalTransformPayload {
> enum Enum {
> PROPERTIES = 0
> [(beam_urn) = "beam:external:transform:external_transform:v1"];
> }
> // A fully-qualified identifier, e.g. Java package + class
> string identifier = 1;
I'd rather the identifier have semantic rather than
implementation-specific meaning. e.g. one could imagine multiple
implementations of a given transform that different services could
offer.
> // the format may change to map<string, bytes> if types are supported
> map<string, string> parameters = 2;
> }
>
> The identifier could also be a URN.
>
> > Can we change first version to map<string, bytes> ? Otherwise the set of transforms we can support/test will be very limited.
>
> How do we do that? Do we define a set of standard coders for supported
> types? On the Java side we can lookup the coder by extracting the field
> from the Pojo, but we can't do that in Python.
>
> > Can we re-use some of the Beam schemas-related work/utilities here ?
>
> Yes, that was the plan.
On this note, Reuven, what is the plan (and timeline) for a
language-independent representation of schemas? The crux of the
problem is that the user needs to specify some kind of configuration
(call it C) to construct the transform (call it T). This would be
handled by a TransformBuilder<C, T> that provides (at least) a mapping
C -> T. (Possibly this interface could be offered on the transform
itself).
The question we are trying to answer here is how to represent C, in
both the source and target language, and on the wire. The idea is that
we could leverage the schema infrastructure such that C could be a
POJO in Java (and perhaps a dict in Python). We would want to extend
Schemas and Row (or perhaps a sub/super/sibling class thereof) to
allow for Coder and UDF-typed fields. (Exactly how to represent UDFs
is still very TBD.) The payload for a external transform using this
format would be the tuple (schema, SchemaCoder(schema).encode(C)). The
goal is to not, yet again, invent a cross-language way of defining a
bag of named, typed parameters (aka fields) with language-idiomatic
mappings and some introspection capabilities, and significantly less
heavy-weight than users defining their own protos (plus generating
bindings to all languages).
Does this seem a reasonable use of schemas?
Re: Cross-language transform API
Posted by Maximilian Michels <mx...@apache.org>.
> Just to clarify. What's the reason for including a PROPERTIES enum here instead of directly making beam_urn a field of ExternalTransformPayload ?
The URN is supposed to be static. We always use the same URN for this
type of external transform. We probably want an additional identifier to
point to the resource we want to configure. Like so:
message ExternalTransformPayload {
enum Enum {
PROPERTIES = 0
[(beam_urn) = "beam:external:transform:external_transform:v1"];
}
// A fully-qualified identifier, e.g. Java package + class
string identifier = 1;
// the format may change to map<string, bytes> if types are supported
map<string, string> parameters = 2;
}
The identifier could also be a URN.
> Can we change first version to map<string, bytes> ? Otherwise the set of transforms we can support/test will be very limited.
How do we do that? Do we define a set of standard coders for supported
types? On the Java side we can lookup the coder by extracting the field
from the Pojo, but we can't do that in Python.
> Can we re-use some of the Beam schemas-related work/utilities here ?
Yes, that was the plan.
-Max
On 11.03.19 16:16, Chamikara Jayalath wrote:
>
>
> On Mon, Mar 11, 2019 at 7:13 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi Cham,
>
> Thank you for the summary of our discussion. For reference I just
> wanted
> to mention that, in addition to the design document [1], there is a
> pending PR [2]. Both need to be adapted to reflect the latest state of
> the discussed design.
>
> Let me quickly recap:
>
> The goal of the PR is to expose the GenerateSequence transform from the
> Java SDK to other SDKs. The current version of the PR uses a
> transform-specific Proto message, i.e. GenerateSequencePayload [3]. The
> message is constructed by the authoring SDK and then sent to the
> ExpansionService which passes on the payload to a builder in
> GenerateSequence.
>
> This approach requires a) a transform-specific payload b)
> transform-specific wrappers in the authoring SDK c) transform-specific
> builders in the external SDK. We decided that it should be easier for
> transform authors to expose their transforms to other SDKs by just
> defining an interface directly in the transform. This very much goes
> back to an earlier version of the discussion in this thread where we
> proposed a JSON builder interface to pass parameters to transforms. We
> realized that this will be sufficient for most use cases, but we wanted
> the schema to be provided by the transform author.
>
> Changes to the PR:
>
> 1) Remove GenerateSequencePayload
> 2) Add generic ExternalTransformPayload, for example:
>
>
> +1 for the general approach here.
>
>
> message ExternalTransformPayload {
> enum Enum {
> PROPERTIES = 0
> [(beam_urn) =
> "beam:external:transform:external_transform:v1"];
> }
>
>
> Just to clarify. What's the reason for including a PROPERTIES enum here
> instead of directly making beam_urn a field of ExternalTransformPayload ?
>
> // the format may change to map<string, bytes> if types are
> supported
> map<string, string> parameters = 1;
> }
>
>
> Can we change first version to map<string, bytes> ? Otherwise the set of
> transforms we can support/test will be very limited.
>
>
> I think we haven't fully figured out the wire format, but we have
> agreed
> that it is a list of parameters which use a predefined set of data
> types. For the scope of the PR, I'll stick with strings.
>
> 3) Define a Pojo in GenerateSequence with the parameters
> 4) Add a schema-based Pojo <=> ExternalTransformPayload conversion
>
>
> Can we re-use some of the Beam schemas-related work/utilities here ?
>
> Thanks,
> Cham
>
> 5) Let Python wrappers use ExternalTransformPayload for the
> request to
> the ExpansionService
>
>
> Once this is done, we should be able to expose any Java transform by
> defining the above-mentioned Pojo. Let me know if I missed anything.
>
>
> Thanks,
> Max
>
> [1] https://s.apache.org/beam-cross-language-io
> [2] https://github.com/apache/beam/pull/7875
> [3] message GenerateSequencePayload {
> enum Enum {
> PROPERTIES = 0 [(beam_urn) =
> "beam:external:transform:generate_sequence:v1"];
> }
> int64 start = 1;
> oneof stop {
> bool stop_provided = 2;
> int64 stop_value = 3;
> }
> oneof elements_per_period {
> bool elements_per_period_provided = 4;
> int64 elements_per_period_value = 5;
> }
> google.protobuf.Duration period = 6;
> google.protobuf.Duration max_read_time = 7;
> }
>
> On 05.03.19 00:28, Chamikara Jayalath wrote:
> > (former title was "Beam Python streaming pipeline on Flink Runner")
> >
> > Hi All,
> >
> > We have some comment threads in the doc and had a couple of offline
> > meetings on the matters discussed in this thread. Here's a
> summary of
> > the discussions so far.
> >
> > For a pipeline defined in SDK A to use a transform T defined in
> SDK B,
> > we need following.
> >
> > (1) Extensions that will allow SDK B (or a service provided by
> SDK B) to
> > instantiate and expand the transform with a given set of parameters.
> >
> > We are gravitating towards (at least as the first version)
> defining a
> > TransformBuilder interface that will take an object P (a POJO for
> Java)
> > as a parameter. Author of transform T will be implementing this
> > interface. SDK B will inspect type of P and will generate a
> proto/schema
> > that will serve as a language-neutral representation of it.
> Probably it
> > should be enough to simply define a proto that is a single Map of
> > strings to objects where strings define the parameters and objects
> > define corresponding values that should be used for parameters.
> We might
> > be able to re-use some of the utilities developed for Beam schema
> > feature for generating a schema from a given POJO.
> >
> > (2) A wire format to plumb through parameters defined in SDK A to
> SDK B.
> >
> > If we use a proto Map to define the interface this should be
> > straightforward. Other options are to use JSON, a more generic proto
> > definition, or a blob of bytes (which is fully passed by a transform
> > builder).
> >
> > (3) Transform stubs in SDK A so that pipeline authors can refer to
> > transform T when defining the pipeline.
> >
> > Probably we should be able to define T in SDK A using a generic
> > transform that takes a URN and a payload that is a set of key,value
> > pairs (a dictionary for Python) as parameters. During transform
> > expansion, this will be converted to proper wire format and will
> be sent
> > to a transform expansion service offered by SDK B. SDK B will use
> this
> > information to build a payload of type P, will instantiate T, will
> > expand it, and will return the result of the expansion to SDK A.
> SDK A
> > will include the expanded definition of T in the job description
> which
> > will be submitted to a runner for execution. Author of transform T
> > could manually define a wrapper API that provides a nicer
> > language-specific API for T for SDK A if needed though these will
> have
> > to be kept manually up-to date to match with interface of T in
> SDK B.
> > In the future we can look into automatically generating nicer
> stubs for
> > remote transforms as a a part of Beam release process.
> >
> > Thanks,
> > Cham
> >
> > On Thu, Feb 14, 2019 at 8:06 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > I've revised the document and included your feedback:
> > https://s.apache.org/beam-cross-language-io
> >
> > I think it reads much better now. I moved away from the JSON
> > configuration in favor of an explicit Proto-based configuration
> > approach
> > which leaves it up to the transform what to include in the Proto
> > payload. Transforms could still be using JSON if it is feasible.
> >
> > Let me know what you think.
> >
> > -Max
> >
> > On 08.02.19 16:07, Robert Bradshaw wrote:
> > > On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Thank you for your comments. They will help to iterate
> over
> > the ideas.
> > >
> > > > I'd like to point out that though this seems to be
> > specifically
> > > targeting IOs, there's nothing here that is specific
> to IOs.
> > >
> > > I thought about this from an IO perspective but I
> agree that
> > it amounts
> > > to using any type of cross-language transform.
> However, there
> > is the IO
> > > specific question of how to parameterize IO transforms.
> > >
> > >
> > > On the other hand, I don't think paramaterizing IO
> transforms is
> > > different than parameterizing any other kind of transform.
> > >
> > > If we end up
> > > using PipelineOptions for that, there would really be no
> > difference
> > > anymore.
> > >
> > >
> > > FWIW, I'm not suggesting that we use the pipeline-level
> > PipelineOptions
> > > (as this makes it impossible to configure the same transform
> > differently
> > > in different parts of the transform). Rather, this is an
> existing
> > > cross-langauge configuration object/pattern we could possibly
> > leverage.
> > > (JSON is much easier to provide from Python (and likely other
> > > languages), but it's harder to know how to read it from
> the Java
> > side,
> > > especially if it has UDFs, coders, schemas, etc. embedded.)
> > >
> > > @Ismael Will add a link after another review round.
> > >
> > > On 08.02.19 09:25, Robert Bradshaw wrote:
> > > > Thanks for writing this up. I'd like to point out
> that though
> > > this seems
> > > > to be specifically targeting IOs, there's nothing
> here that is
> > > specific
> > > > to IOs.
> > > >
> > > > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath
> > > <chamikara@google.com <ma...@google.com>
> <mailto:chamikara@google.com <ma...@google.com>>
> > <mailto:chamikara@google.com <ma...@google.com>
> <mailto:chamikara@google.com <ma...@google.com>>>
> > > > <mailto:chamikara@google.com
> <ma...@google.com> <mailto:chamikara@google.com
> <ma...@google.com>>
> > <mailto:chamikara@google.com <ma...@google.com>
> <mailto:chamikara@google.com <ma...@google.com>>>>> wrote:
> > > >
> > > > Thanks Max. Added few comments.
> > > >
> > > > - Cham
> > > >
> > > > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía
> > > <iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>
> > > > <mailto:iemejia@gmail.com
> <ma...@gmail.com> <mailto:iemejia@gmail.com
> <ma...@gmail.com>>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>>> wrote:
> > > >
> > > > Can you please add the link to the design
> document
> > webpage.
> > > >
> > > > Le jeu. 7 févr. 2019 à 19:59, Maximilian
> Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> a écrit :
> > > >
> > > > I've created an initial design document:
> > > > https://s.apache.org/beam-cross-language-io
> > > >
> > > > It does not contain all the details but
> > perhaps it's
> > > a good
> > > > basis for a
> > > > discussion on how we proceed.
> > > >
> > > > -Max
> > > >
> > > > On 06.02.19 19:49, Chamikara Jayalath
> wrote:
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 8:38 AM
> Maximilian
> > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > Thanks for your replies Robert
> and Cham.
> > > > >
> > > > > What I had in mind was a generic
> > Wrapper that
> > > would
> > > > easily allow users
> > > > > to use IO from Java. Such
> wrapper could
> > start
> > > as an
> > > > experimental
> > > > > feature
> > > > > and then, through URN versioning,
> > become stable
> > > > eventually.
> > > > >
> > > > > UDFs are needed, though they are
> a special
> > > case. Most
> > > > users (including
> > > > > Matthias) just want to specify a
> few String
> > > options
> > > > which do not
> > > > > require
> > > > > UDFs but something along the
> lines what I
> > > proposed here.
> > > > >
> > > > >
> > > > > Sounds good let's start
> > documenting/implementing the
> > > > "easy" case and
> > > > > think bit more regarding UDFs.
> > > > >
> > > > >
> > > > >
> > > > > Robert wrote:
> > > > > > UDFs that are called from
> within an
> > IO as
> > > part of
> > > > its operation is
> > > > > > still an open question.
> > > > >
> > > > > Exactly. How about we solve the
> easier
> > case first,
> > > > unblock users, and
> > > > > then think more about solving the
> > general case?
> > > > >
> > > > > Cham wrote:
> > > > > > I'm happy to work with you to
> > realize this.
> > > > >
> > > > > Would be great to exchange more
> ideas
> > on this!
> > > I can
> > > > compile the
> > > > > current
> > > > > ideas we have in a document and
> we move
> > from
> > > there.
> > > > >
> > > > >
> > > > > Great. Thanks.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Max
> > > > >
> > > > > On 05.02.19 17:56, Chamikara
> Jayalath
> > wrote:
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 5, 2019, 8:11 AM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > Good points Cham.
> > > > > >
> > > > > > JSON seemed like the most
> > intuitive way to
> > > > specify a
> > > > > configuration map.
> > > > > > We already use JSON in
> other places,
> > > e.g. to
> > > > specify the
> > > > > environment
> > > > > > configuration. It is not
> > necessarily a
> > > > contradiction to have JSON
> > > > > > inside
> > > > > > Protobuf. From the
> perspective of IO
> > > authors, the
> > > > > user-friendliness
> > > > > > plays a role because they
> > wouldn't have to
> > > > deal with Protobuf.
> > > > > >
> > > > > >
> > > > > > It's a good point that JSON will
> > make this more
> > > > user-friendly for IO
> > > > > > transforms authors. Probably we
> > should do a
> > > bit of
> > > > > experimentation and
> > > > > > keep this experimental in
> case we hit a
> > > > performance snag.
> > > > > >
> > > > > >
> > > > > > I agree that the
> configuration
> > format is an
> > > > implementation
> > > > > detail that
> > > > > > will be hidden to users via
> > easy-to-use
> > > wrappers.
> > > > > >
> > > > > > Do we have to support
> UDFs for
> > expanding
> > > > existing IO? Users
> > > > > would still
> > > > > > be able to apply UDFs via
> ParDo
> > on the IO
> > > > output collections.
> > > > > Generally
> > > > > > speaking, I can see how for
> > cross-language
> > > > transforms UDF
> > > > > support would
> > > > > > be good. For example, a
> Combine
> > > implementation
> > > > in Java, where the
> > > > > > combine UDFs come from
> Python.
> > > > > >
> > > > > >
> > > > > > I think we should try to
> support UDFs in
> > > the first
> > > > version unless
> > > > > > there's a major blocker that
> hinders
> > realizing
> > > > this. Many IO
> > > > > transforms
> > > > > > available today expect users
> to pass
> > UDFs to
> > > > realize various
> > > > > features
> > > > > > (for example, dynamic
> destinations for
> > > BigqueryIO
> > > > and FileIO,
> > > > > timestamp
> > > > > > function for KafkaIO). I
> think without
> > > support for
> > > > UDFs usability of
> > > > > > cross-language transforms feature
> > will be
> > > > significantly limited.
> > > > > >
> > > > > > I'm happy to work with you to
> > realize this.
> > > > > >
> > > > > > Thanks,
> > > > > > Cham
> > > > > >
> > > > > >
> > > > > > I suppose the question
> is, do we
> > try to
> > > solve
> > > > the general
> > > > > case, or
> > > > > > do we
> > > > > > go with a simpler
> approach for
> > enabling the
> > > > use of existing
> > > > > IO first?
> > > > > > Lack of IO seems to be
> the most
> > > pressing issue
> > > > for the
> > > > > adoption of Beam
> > > > > > Python. I imagine that a
> > > backwards-compatible
> > > > incremental
> > > > > support for
> > > > > > cross-language transforms
> (IOs
> > first, later
> > > > other transforms)
> > > > > would be
> > > > > > possible.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > > On 05.02.19 03:07, Chamikara
> > Jayalath
> > > wrote:
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 1, 2019 at
> 6:12 AM
> > > Maximilian
> > > > Michels
> > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > >
> > > > > > > Yes, I imagine
> sources to
> > > implement a
> > > > JsonConfigurable
> > > > > > interface (e.g.
> > > > > > > on their builders):
> > > > > > >
> > > > > > > JsonConfigurable {
> > > > > > > // Either a json
> > string or
> > > > Map<String, String>
> > > > > > > apply(String
> jsonConfig);
> > > > > > > }
> > > > > > >
> > > > > > > In Python we would
> create
> > this
> > > transform:
> > > > > > >
> > > > > > > URN:
> JsonConfiguredSource:v1
> > > > > > > payload: {
> > > > > > > environment:
> > environment_id, //
> > > > Java/Python/Go
> > > > > > >
> resourceIdentifier:
> > string, //
> > > > > > "org.apache.beam.io.PubSubIO"
> > > > > > >
> configuration: json
> > > config, // {
> > > > "topic" :
> > > > > > "my_pubsub_topic" }
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > Thanks Max, this is a
> great
> > first step
> > > > towards defining to
> > > > > API for
> > > > > > > cross-language transforms.
> > > > > > > Is there a reason why you
> > would want
> > > to use
> > > > JSON instead
> > > > > of a proto
> > > > > > > here. I guess we'll be
> > providing a more
> > > > user friendly language
> > > > > > wrapper
> > > > > > > (for example, Python) for
> > end-users
> > > here, so
> > > > > > user-friendliness-wise, the
> > > > > > > format we choose won't
> matter
> > much (for
> > > > pipeline authors).
> > > > > > > If we don't support UDFs,
> > performance
> > > > difference will be
> > > > > > negligible, but
> > > > > > > UDFs might require a
> callback to
> > > original
> > > > SDK (per-element
> > > > > worst
> > > > > > case).
> > > > > > > So might make sense to
> choose
> > the more
> > > > efficient format.
> > > > > > >
> > > > > > > Also, probably we need to
> > define a more
> > > > expanded definition
> > > > > > (proto/JSON)
> > > > > > > to support UDFs. For
> example, a
> > > payload + a
> > > > set of parameter
> > > > > > definitions
> > > > > > > so that the target SDK
> (for
> > example,
> > > Java)
> > > > can call back the
> > > > > > original
> > > > > > > SDK where the pipeline was
> > authored
> > > in (for
> > > > example,
> > > > > Python) to
> > > > > > resolve
> > > > > > > UDFs at runtime.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Cham
> > > > > > >
> > > > > > > That's more
> generic and
> > could be
> > > used
> > > > for other languages
> > > > > > where we
> > > > > > > might
> > > > > > > have sources/sinks.
> > > > > > >
> > > > > > > > (FWIW, I was
> imagining
> > PubSubIO
> > > > already had a
> > > > > translation into
> > > > > > > BeamFnApi protos
> that fully
> > > specified
> > > > it, and we use
> > > > > that same
> > > > > > > format to
> translate back
> > out.)
> > > > > > >
> > > > > > > Not that I know of.
> > > > > > >
> > > > > > > On 01.02.19 14:02,
> Robert
> > > Bradshaw wrote:
> > > > > > > �� > Are you suggesting
> > something
> > > akin to
> > > > a generic
> > > > > > > >
> > > > > > > > urn:
> > > JsonConfiguredJavaSource
> > > > > > > > payload:
> some json
> > > specifying
> > > > which source and
> > > > > which
> > > > > > parameters
> > > > > > > >
> > > > > > > > which would
> expand to
> > actually
> > > > constructing and
> > > > > applying
> > > > > > that source?
> > > > > > > >
> > > > > > > > (FWIW, I was
> imagining
> > PubSubIO
> > > > already had a
> > > > > translation into
> > > > > > > BeamFnApi
> > > > > > > > protos that fully
> > specified
> > > it, and
> > > > we use that
> > > > > same format to
> > > > > > > translate
> > > > > > > > back out.)
> > > > > > > >
> > > > > > > > On Fri, Feb 1,
> 2019 at
> > 1:44 PM
> > > > Maximilian Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > >
> > > > > > > >�� Recaping here:
> > > > > > > >
> > > > > > > > We all
> agree that
> > SDF is
> > > the way
> > > > to go for future
> > > > > > > implementations of
> > > > > > > > sources. It
> > enables us to get
> > > > rid of the source
> > > > > > interfaces.
> > > > > > > However,
> > > > > > > > SDF
> > > > > > > > does not
> solve the
> > lack of
> > > > streaming sources in
> > > > > Python.
> > > > > > > >
> > > > > > > > The
> expansion PR
> > (thanks
> > > btw!)
> > > > solves the
> > > > > problem of
> > > > > > > >
> expanding/translating
> > > URNs known
> > > > to an
> > > > > > ExpansionService. That
> > > > > > > is a more
> > > > > > > >
> programmatic way
> > of replacing
> > > > language-specific
> > > > > performs,
> > > > > > > instead of
> > > > > > > > relying on
> translators
> > > directly
> > > > in the Runner.
> > > > > > > >
> > > > > > > > What is
> unsolved
> > is the
> > > > configuration of
> > > > > sources from
> > > > > > a foreign
> > > > > > > >
> environment. In my
> > > opinion this
> > > > is the most
> > > > > pressing issue
> > > > > > > for Python
> > > > > > > > sources,
> because
> > what is
> > > > PubSubIO worth in
> > > > > Python if
> > > > > > you cannot
> > > > > > > > configure it?
> > > > > > > >
> > > > > > > > What about
> this:
> > > > > > > >
> > > > > > > > I think it is
> > worth adding a
> > > > JSON configuration
> > > > > option for
> > > > > > > all existing
> > > > > > > > Java
> sources. That
> > way,
> > > we could
> > > > easily
> > > > > configure them as
> > > > > > > part of the
> > > > > > > > expansion
> request
> > (which
> > > would
> > > > contain a JSON
> > > > > > configuration).
> > > > > > > I'll
> > > > > > > > probably fork a
> > thread to
> > > > discuss this in more
> > > > > detail,
> > > > > > but would
> > > > > > > > like to
> > > > > > > > hear your
> thoughts.
> > > > > > > >
> > > > > > > > -Max
> > > > > > > >
> > > > > > > > On 01.02.19
> 13:08,
> > Robert
> > > > Bradshaw wrote:
> > > > > > > > > On Thu,
> Jan 31,
> > 2019
> > > at 6:25
> > > > PM Maximilian
> > > > > Michels
> > > > > > > >
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > >
> > > > > > > > > Ah, I
> > thought you
> > > meant
> > > > native Flink
> > > > > transforms.
> > > > > > > > >
> > > > > > > > >
> Exactly! The
> > > translation
> > > > code is already
> > > > > there.
> > > > > > The main
> > > > > > > > challenge
> > > > > > > > > is
> how to
> > > > > > > > >
> > programmatically
> > > > configure the BeamIO from
> > > > > > Python. I
> > > > > > > suppose
> > > > > > > > that is
> > > > > > > > > also an
> > > > > > > > > unsolved
> > problem for
> > > > cross-language
> > > > > transforms
> > > > > > in general.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is what
> > > > > >
> > https://github.com/apache/beam/pull/7316 does.
> > > > > > > > >
> > > > > > > > > For a
> particular
> > > source, one
> > > > would want to
> > > > > define a
> > > > > > URN and
> > > > > > > > >
> corresponding
> > payload,
> > > then
> > > > (probably) a
> > > > > > CompositeTransform in
> > > > > > > > Python
> > > > > > > > > that
> takes the
> > users
> > > > arguments, packages
> > > > > them into the
> > > > > > > payload,
> > > > > > > > applies
> > > > > > > > > the
> > ExternalTransform, and
> > > > returns the
> > > > > results. How
> > > > > > to handle
> > > > > > > > arbitrary
> > > > > > > > > UDFs
> embedded in
> > > sources is
> > > > still TBD.
> > > > > > > > >
> > > > > > > > > For
> > Matthias' pipeline
> > > > with PubSubIO we
> > > > > can build
> > > > > > > something
> > > > > > > > >
> specific,
> > but for
> > > > > > > > > the
> general
> > case there
> > > > should be way to
> > > > > > initialize a
> > > > > > > Beam IO
> > > > > > > > via a
> > > > > > > > >
> configuration
> > > > > > > > > map
> > provided by an
> > > > external environment.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I
> thought quite
> > a bit
> > > about
> > > > how we could
> > > > > represent
> > > > > > expansions
> > > > > > > > statically
> > > > > > > > > (e.g.
> have some
> > kind of
> > > > expansion template that
> > > > > > could be
> > > > > > > used, at
> > > > > > > > least
> > > > > > > > > in many
> cases,
> > as data
> > > > without firing up a
> > > > > separate
> > > > > > > process. May be
> > > > > > > > > worth doing
> > > eventually, but
> > > > we run into the
> > > > > same issues
> > > > > > > that were
> > > > > > > > > discussed at
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> https://github.com/apache/beam/pull/7316#discussion_r249996455
> > > > ).
> > > > > > > > >
> > > > > > > > > If one is
> > already using a
> > > > portable runner
> > > > > like Flink,
> > > > > > > having the job
> > > > > > > > > service
> process
> > > automatically
> > > > also serve up an
> > > > > > expansion
> > > > > > > service for
> > > > > > > > > various
> URNs it
> > knows and
> > > > cares about is
> > > > > probably a
> > > > > > pretty
> > > > > > > low bar.
> > > > > > > > > Flink could
> > serve up
> > > things
> > > > it would rather
> > > > > get back
> > > > > > > untouched in a
> > > > > > > > >
> transform with
> > a special
> > > > flink runner urn.
> > > > > > > > >
> > > > > > > > > As Ahmet
> > mentions, SDF is
> > > > better solution. I
> > > > > hope
> > > > > > it's not
> > > > > > > that far
> > > > > > > > > away,
> but even
> > once it
> > > comes
> > > > we'll likely
> > > > > want the
> > > > > > above
> > > > > > > > framework to
> > > > > > > > > invoke
> the full
> > suite
> > > of Java
> > > > IOs even after
> > > > > they're
> > > > > > > running on SDF
> > > > > > > > > themselves.
> > > > > > > > >
> > > > > > > > > - Robert
> > > > > > > > >
> > > > > > > > > On
> 31.01.19
> > 17:36,
> > > Thomas
> > > > Weise wrote:
> > > > > > > > > >
> Exactly,
> > that's
> > > what I
> > > > had in mind.
> > > > > > > > > >
> > > > > > > > > > A
> Flink
> > runner
> > > native
> > > > transform would
> > > > > make
> > > > > > the existing
> > > > > > > > unbounded
> > > > > > > > > sources
> > > > > > > > > >
> available,
> > > similar to:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> On Thu,
> > Jan 31,
> > > 2019
> > > > at 8:18 AM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>> wrote:
> > > > > > > > > >
> > > > > > > > > >
> > Wouldn't it be
> > > > even more useful
> > > > > for the
> > > > > > transition
> > > > > > > > period if
> > > > > > > > > we
> enabled
> > Beam IO
> > > > > > > > > >
> to
> > be used via
> > > > Flink (like in the
> > > > > legacy
> > > > > > Flink
> > > > > > > Runner)? In
> > > > > > > > > this
> particular
> > > > > > > > > >
> example,
> > > Matthias
> > > > wants to use
> > > > > PubSubIO,
> > > > > > which
> > > > > > > is not even
> > > > > > > > >
> available as a
> > > > > > > > > >
> native
> > > > > > > > > >
> Flink
> > > transform.
> > > > > > > > > >
> > > > > > > > > >
> On
> > 31.01.19
> > > 16:21,
> > > > Thomas Weise
> > > > > wrote:
> > > > > > > > > >
> >
> > Until SDF is
> > > > supported, we
> > > > > could also add
> > > > > > > Flink runner
> > > > > > > > > native
> > transforms for
> > > > > > > > > >
> >
> > selected
> > > > unbounded sources [1].
> > > > > > > > > >
> >
> > > > > > > > > >
> > That
> > > might be a
> > > > reasonable
> > > > > option to
> > > > > > unblock
> > > > > > > users that
> > > > > > > > > want
> to try
> > Python
> > > > > > > > > >
> >
> > > streaming on Flink.
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > Thomas
> > > > > > > > > >
> >
> > > > > > > > > >
> > [1]
> > > > > > > > > >
> >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > On Thu,
> > > Jan 31,
> > > > 2019 at 6:51 AM
> > > > > > Maximilian
> > > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>
> > > > > > > > > >
> >
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > ��<mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>>> wrote:
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > > I
> > > have a
> > > > hard time to
> > > > > imagine
> > > > > > how can
> > > > > > > we map
> > > > > > > > in a
> > > > > > > > >
> generic way
> > > > > > > > > >
> >
> > > > RestrictionTrackers into
> > > > > the existing
> > > > > > > > >
> > > Bounded/UnboundedSource,
> > > > so I would
> > > > > > > > > >
> >
> > love to
> > > > hear more about
> > > > > the details.
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > Isn't it
> > > > the other way around?
> > > > > > The SDF is a
> > > > > > > > >
> > generalization of
> > > > > > > > > >
> > > UnboundedSource.
> > > > > > > > > >
> >
> > So
> > > we would
> > > > wrap
> > > > > UnboundedSource
> > > > > > using
> > > > > > > SDF. I'm not
> > > > > > > > >
> saying it is
> > > > > > > > > >
> > trivial, but
> > > > > > > > > >
> >
> > SDF
> > > offers
> > > > all the
> > > > > functionality that
> > > > > > > > UnboundedSource
> > > > > > > > > needs.
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > For
> > > > example, the
> > > > > > @GetInitialRestriction
> > > > > > > method
> > > > > > > > would
> > > > > > > > > call
> split
> > on the
> > > > > > > > > >
> >
> > > > UnboundedSource and the
> > > > > restriction
> > > > > > > trackers would
> > > > > > > > > then
> be used to
> > > > > > > > > >
> > process the
> > > > > > > > > >
> >
> > splits.
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > On
> > > 31.01.19
> > > > 15:16, Ismaël
> > > > > Mejía
> > > > > > wrote:
> > > > > > > > > >
> >
> > >> Not
> > > > necessarily. This
> > > > > would
> > > > > > be one
> > > > > > > way. Another
> > > > > > > > > way is
> > build an SDF
> > > > > > > > > >
> >
> > > wrapper for
> > > > UnboundedSource.
> > > > > > Probably the
> > > > > > > > easier path
> > > > > > > > > for
> migration.
> > > > > > > > > >
> >
> > >
> > > > > > > > > >
> >
> > > That
> > > > would be fantastic, I
> > > > > > have heard
> > > > > > > about such
> > > > > > > > > wrapper
> > multiple
> > > > > > > > > >
> >
> > > times
> > > > but so far there
> > > > > is not any
> > > > > > > realistic
> > > > > > > > >
> proposal. I
> > have a
> > > hard
> > > > > > > > > >
> >
> > >
> > > time to
> > > > imagine how can
> > > > > we map
> > > > > > in a
> > > > > > > generic way
> > > > > > > > >
> > RestrictionTrackers
> > > > > > > > > >
> >
> > >
> > > into the
> > > > existing
> > > > > > >
> Bounded/UnboundedSource, so I
> > > > > > > > >
> would love
> > to hear
> > > > > > > > > >
> >
> > > more
> > > > about the details.
> > > > > > > > > >
> >
> > >
> > > > > > > > > >
> >
> > >
> > > On Thu,
> > > > Jan 31, 2019 at
> > > > > 3:07 PM
> > > > > > > Maximilian
> > > > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>
> > > > > > > > > >
> >
> > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>>> wrote:
> > > > > > > > > >
> >
> > >>
> > > > > > > > > >
> >
> > >>
> > > > In
> > > > addition to have
> > > > > > support in
> > > > > > > the runners,
> > > > > > > > > this
> will
> > require a
> > > > > > > > > >
> >
> > >> >
> > > > rewrite of PubsubIO to
> > > > > > use the
> > > > > > > new SDF API.
> > > > > > > > > >
> >
> > >>
> > > > > > > > > >
> >
> > >> Not
> > > > necessarily. This
> > > > > would
> > > > > > be one
> > > > > > > way. Another
> > > > > > > > > way is
> > build an SDF
> > > > > > > > > >
> >
> > > wrapper for
> > > > > > > > > >
> >
> > >>
> > > > UnboundedSource.
> > > > > Probably the
> > > > > > easier
> > > > > > > path for
> > > > > > > > >
> migration.
> > > > > > > > > >
> >
> > >>
> > > > > > > > > >
> >
> > >> On
> > > > 31.01.19 14:03, Ismaël
> > > > > > Mejía wrote:
> > > > > > > > > >
> >
> > >>>>
> > > > Fortunately, there
> > > > > is already a
> > > > > > > pending
> > > > > > > > PR for
> > > > > > > > >
> cross-language
> > > > > > > > > >
> >
> > > pipelines which
> > > > > > > > > >
> >
> > > >>>> will
> > > > allow us to use
> > > > > Java
> > > > > > IO like
> > > > > > > PubSub in
> > > > > > > > >
> Python jobs.
> > > > > > > > > >
> >
> > >>>
> > > > > > > > > >
> >
> > >>> In
> > > > addition to have
> > > > > support
> > > > > > in the
> > > > > > > > runners, this
> > > > > > > > > will
> require a
> > > > > > > > > >
> >
> > >>>
> > > > rewrite of PubsubIO
> > > > > to use
> > > > > > the new
> > > > > > > SDF API.
> > > > > > > > > >
> >
> > >>>
> > > > > > > > > >
> >
> > >>> On
> > > > Thu, Jan 31, 2019
> > > > > at 12:23 PM
> > > > > > > > Maximilian
> Michels
> > > > > > > > > >
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>
> > > > > > > > > >
> >
> > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > <mail
> > > >
> > >
> >
>
Re: Cross-language transform API
Posted by Chamikara Jayalath <ch...@google.com>.
On Mon, Mar 11, 2019 at 7:13 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Cham,
>
> Thank you for the summary of our discussion. For reference I just wanted
> to mention that, in addition to the design document [1], there is a
> pending PR [2]. Both need to be adapted to reflect the latest state of
> the discussed design.
>
> Let me quickly recap:
>
> The goal of the PR is to expose the GenerateSequence transform from the
> Java SDK to other SDKs. The current version of the PR uses a
> transform-specific Proto message, i.e. GenerateSequencePayload [3]. The
> message is constructed by the authoring SDK and then sent to the
> ExpansionService which passes on the payload to a builder in
> GenerateSequence.
>
> This approach requires a) a transform-specific payload b)
> transform-specific wrappers in the authoring SDK c) transform-specific
> builders in the external SDK. We decided that it should be easier for
> transform authors to expose their transforms to other SDKs by just
> defining an interface directly in the transform. This very much goes
> back to an earlier version of the discussion in this thread where we
> proposed a JSON builder interface to pass parameters to transforms. We
> realized that this will be sufficient for most use cases, but we wanted
> the schema to be provided by the transform author.
>
> Changes to the PR:
>
> 1) Remove GenerateSequencePayload
> 2) Add generic ExternalTransformPayload, for example:
>
+1 for the general approach here.
>
> message ExternalTransformPayload {
> enum Enum {
> PROPERTIES = 0
> [(beam_urn) = "beam:external:transform:external_transform:v1"];
> }
>
Just to clarify. What's the reason for including a PROPERTIES enum here
instead of directly making beam_urn a field of ExternalTransformPayload ?
> // the format may change to map<string, bytes> if types are supported
> map<string, string> parameters = 1;
> }
>
Can we change first version to map<string, bytes> ? Otherwise the set of
transforms we can support/test will be very limited.
>
> I think we haven't fully figured out the wire format, but we have agreed
> that it is a list of parameters which use a predefined set of data
> types. For the scope of the PR, I'll stick with strings.
>
> 3) Define a Pojo in GenerateSequence with the parameters
> 4) Add a schema-based Pojo <=> ExternalTransformPayload conversion
>
Can we re-use some of the Beam schemas-related work/utilities here ?
Thanks,
Cham
> 5) Let Python wrappers use ExternalTransformPayload for the request to
> the ExpansionService
>
>
> Once this is done, we should be able to expose any Java transform by
> defining the above-mentioned Pojo. Let me know if I missed anything.
>
>
> Thanks,
> Max
>
> [1] https://s.apache.org/beam-cross-language-io
> [2] https://github.com/apache/beam/pull/7875
> [3] message GenerateSequencePayload {
> enum Enum {
> PROPERTIES = 0 [(beam_urn) =
> "beam:external:transform:generate_sequence:v1"];
> }
> int64 start = 1;
> oneof stop {
> bool stop_provided = 2;
> int64 stop_value = 3;
> }
> oneof elements_per_period {
> bool elements_per_period_provided = 4;
> int64 elements_per_period_value = 5;
> }
> google.protobuf.Duration period = 6;
> google.protobuf.Duration max_read_time = 7;
> }
>
> On 05.03.19 00:28, Chamikara Jayalath wrote:
> > (former title was "Beam Python streaming pipeline on Flink Runner")
> >
> > Hi All,
> >
> > We have some comment threads in the doc and had a couple of offline
> > meetings on the matters discussed in this thread. Here's a summary of
> > the discussions so far.
> >
> > For a pipeline defined in SDK A to use a transform T defined in SDK B,
> > we need following.
> >
> > (1) Extensions that will allow SDK B (or a service provided by SDK B) to
> > instantiate and expand the transform with a given set of parameters.
> >
> > We are gravitating towards (at least as the first version) defining a
> > TransformBuilder interface that will take an object P (a POJO for Java)
> > as a parameter. Author of transform T will be implementing this
> > interface. SDK B will inspect type of P and will generate a proto/schema
> > that will serve as a language-neutral representation of it. Probably it
> > should be enough to simply define a proto that is a single Map of
> > strings to objects where strings define the parameters and objects
> > define corresponding values that should be used for parameters. We might
> > be able to re-use some of the utilities developed for Beam schema
> > feature for generating a schema from a given POJO.
> >
> > (2) A wire format to plumb through parameters defined in SDK A to SDK B.
> >
> > If we use a proto Map to define the interface this should be
> > straightforward. Other options are to use JSON, a more generic proto
> > definition, or a blob of bytes (which is fully passed by a transform
> > builder).
> >
> > (3) Transform stubs in SDK A so that pipeline authors can refer to
> > transform T when defining the pipeline.
> >
> > Probably we should be able to define T in SDK A using a generic
> > transform that takes a URN and a payload that is a set of key,value
> > pairs (a dictionary for Python) as parameters. During transform
> > expansion, this will be converted to proper wire format and will be sent
> > to a transform expansion service offered by SDK B. SDK B will use this
> > information to build a payload of type P, will instantiate T, will
> > expand it, and will return the result of the expansion to SDK A. SDK A
> > will include the expanded definition of T in the job description which
> > will be submitted to a runner for execution. Author of transform T
> > could manually define a wrapper API that provides a nicer
> > language-specific API for T for SDK A if needed though these will have
> > to be kept manually up-to date to match with interface of T in SDK B.
> > In the future we can look into automatically generating nicer stubs for
> > remote transforms as a a part of Beam release process.
> >
> > Thanks,
> > Cham
> >
> > On Thu, Feb 14, 2019 at 8:06 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > I've revised the document and included your feedback:
> > https://s.apache.org/beam-cross-language-io
> >
> > I think it reads much better now. I moved away from the JSON
> > configuration in favor of an explicit Proto-based configuration
> > approach
> > which leaves it up to the transform what to include in the Proto
> > payload. Transforms could still be using JSON if it is feasible.
> >
> > Let me know what you think.
> >
> > -Max
> >
> > On 08.02.19 16:07, Robert Bradshaw wrote:
> > > On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Thank you for your comments. They will help to iterate over
> > the ideas.
> > >
> > > > I'd like to point out that though this seems to be
> > specifically
> > > targeting IOs, there's nothing here that is specific to IOs.
> > >
> > > I thought about this from an IO perspective but I agree that
> > it amounts
> > > to using any type of cross-language transform. However, there
> > is the IO
> > > specific question of how to parameterize IO transforms.
> > >
> > >
> > > On the other hand, I don't think paramaterizing IO transforms is
> > > different than parameterizing any other kind of transform.
> > >
> > > If we end up
> > > using PipelineOptions for that, there would really be no
> > difference
> > > anymore.
> > >
> > >
> > > FWIW, I'm not suggesting that we use the pipeline-level
> > PipelineOptions
> > > (as this makes it impossible to configure the same transform
> > differently
> > > in different parts of the transform). Rather, this is an existing
> > > cross-langauge configuration object/pattern we could possibly
> > leverage.
> > > (JSON is much easier to provide from Python (and likely other
> > > languages), but it's harder to know how to read it from the Java
> > side,
> > > especially if it has UDFs, coders, schemas, etc. embedded.)
> > >
> > > @Ismael Will add a link after another review round.
> > >
> > > On 08.02.19 09:25, Robert Bradshaw wrote:
> > > > Thanks for writing this up. I'd like to point out that
> though
> > > this seems
> > > > to be specifically targeting IOs, there's nothing here
> that is
> > > specific
> > > > to IOs.
> > > >
> > > > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath
> > > <chamikara@google.com <ma...@google.com>
> > <mailto:chamikara@google.com <ma...@google.com>>
> > > > <mailto:chamikara@google.com <ma...@google.com>
> > <mailto:chamikara@google.com <ma...@google.com>>>> wrote:
> > > >
> > > > Thanks Max. Added few comments.
> > > >
> > > > - Cham
> > > >
> > > > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía
> > > <iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > > > <mailto:iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>>> wrote:
> > > >
> > > > Can you please add the link to the design document
> > webpage.
> > > >
> > > > Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> a écrit :
> > > >
> > > > I've created an initial design document:
> > > > https://s.apache.org/beam-cross-language-io
> > > >
> > > > It does not contain all the details but
> > perhaps it's
> > > a good
> > > > basis for a
> > > > discussion on how we proceed.
> > > >
> > > > -Max
> > > >
> > > > On 06.02.19 19:49, Chamikara Jayalath wrote:
> > > > >
> > > > >
> > > > > On Wed, Feb 6, 2019 at 8:38 AM Maximilian
> > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Thanks for your replies Robert and Cham.
> > > > >
> > > > > What I had in mind was a generic
> > Wrapper that
> > > would
> > > > easily allow users
> > > > > to use IO from Java. Such wrapper could
> > start
> > > as an
> > > > experimental
> > > > > feature
> > > > > and then, through URN versioning,
> > become stable
> > > > eventually.
> > > > >
> > > > > UDFs are needed, though they are a
> special
> > > case. Most
> > > > users (including
> > > > > Matthias) just want to specify a few
> String
> > > options
> > > > which do not
> > > > > require
> > > > > UDFs but something along the lines what
> I
> > > proposed here.
> > > > >
> > > > >
> > > > > Sounds good let's start
> > documenting/implementing the
> > > > "easy" case and
> > > > > think bit more regarding UDFs.
> > > > >
> > > > >
> > > > >
> > > > > Robert wrote:
> > > > > > UDFs that are called from within an
> > IO as
> > > part of
> > > > its operation is
> > > > > > still an open question.
> > > > >
> > > > > Exactly. How about we solve the easier
> > case first,
> > > > unblock users, and
> > > > > then think more about solving the
> > general case?
> > > > >
> > > > > Cham wrote:
> > > > > > I'm happy to work with you to
> > realize this.
> > > > >
> > > > > Would be great to exchange more ideas
> > on this!
> > > I can
> > > > compile the
> > > > > current
> > > > > ideas we have in a document and we move
> > from
> > > there.
> > > > >
> > > > >
> > > > > Great. Thanks.
> > > > >
> > > > >
> > > > > Thanks,
> > > > > Max
> > > > >
> > > > > On 05.02.19 17:56, Chamikara Jayalath
> > wrote:
> > > > > >
> > > > > >
> > > > > > On Tue, Feb 5, 2019, 8:11 AM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > > >
> > > > > > Good points Cham.
> > > > > >
> > > > > > JSON seemed like the most
> > intuitive way to
> > > > specify a
> > > > > configuration map.
> > > > > > We already use JSON in other
> places,
> > > e.g. to
> > > > specify the
> > > > > environment
> > > > > > configuration. It is not
> > necessarily a
> > > > contradiction to have JSON
> > > > > > inside
> > > > > > Protobuf. From the perspective
> of IO
> > > authors, the
> > > > > user-friendliness
> > > > > > plays a role because they
> > wouldn't have to
> > > > deal with Protobuf.
> > > > > >
> > > > > >
> > > > > > It's a good point that JSON will
> > make this more
> > > > user-friendly for IO
> > > > > > transforms authors. Probably we
> > should do a
> > > bit of
> > > > > experimentation and
> > > > > > keep this experimental in case we
> hit a
> > > > performance snag.
> > > > > >
> > > > > >
> > > > > > I agree that the configuration
> > format is an
> > > > implementation
> > > > > detail that
> > > > > > will be hidden to users via
> > easy-to-use
> > > wrappers.
> > > > > >
> > > > > > Do we have to support UDFs for
> > expanding
> > > > existing IO? Users
> > > > > would still
> > > > > > be able to apply UDFs via ParDo
> > on the IO
> > > > output collections.
> > > > > Generally
> > > > > > speaking, I can see how for
> > cross-language
> > > > transforms UDF
> > > > > support would
> > > > > > be good. For example, a Combine
> > > implementation
> > > > in Java, where the
> > > > > > combine UDFs come from Python.
> > > > > >
> > > > > >
> > > > > > I think we should try to support
> UDFs in
> > > the first
> > > > version unless
> > > > > > there's a major blocker that hinders
> > realizing
> > > > this. Many IO
> > > > > transforms
> > > > > > available today expect users to pass
> > UDFs to
> > > > realize various
> > > > > features
> > > > > > (for example, dynamic destinations
> for
> > > BigqueryIO
> > > > and FileIO,
> > > > > timestamp
> > > > > > function for KafkaIO). I think
> without
> > > support for
> > > > UDFs usability of
> > > > > > cross-language transforms feature
> > will be
> > > > significantly limited.
> > > > > >
> > > > > > I'm happy to work with you to
> > realize this.
> > > > > >
> > > > > > Thanks,
> > > > > > Cham
> > > > > >
> > > > > >
> > > > > > I suppose the question is, do we
> > try to
> > > solve
> > > > the general
> > > > > case, or
> > > > > > do we
> > > > > > go with a simpler approach for
> > enabling the
> > > > use of existing
> > > > > IO first?
> > > > > > Lack of IO seems to be the most
> > > pressing issue
> > > > for the
> > > > > adoption of Beam
> > > > > > Python. I imagine that a
> > > backwards-compatible
> > > > incremental
> > > > > support for
> > > > > > cross-language transforms (IOs
> > first, later
> > > > other transforms)
> > > > > would be
> > > > > > possible.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > > On 05.02.19 03:07, Chamikara
> > Jayalath
> > > wrote:
> > > > > > >
> > > > > > >
> > > > > > > On Fri, Feb 1, 2019 at 6:12 AM
> > > Maximilian
> > > > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >
> > > > > > > Yes, I imagine sources to
> > > implement a
> > > > JsonConfigurable
> > > > > > interface (e.g.
> > > > > > > on their builders):
> > > > > > >
> > > > > > > JsonConfigurable {
> > > > > > > // Either a json
> > string or
> > > > Map<String, String>
> > > > > > > apply(String
> jsonConfig);
> > > > > > > }
> > > > > > >
> > > > > > > In Python we would create
> > this
> > > transform:
> > > > > > >
> > > > > > > URN:
> JsonConfiguredSource:v1
> > > > > > > payload: {
> > > > > > > environment:
> > environment_id, //
> > > > Java/Python/Go
> > > > > > > resourceIdentifier:
> > string, //
> > > > > > "org.apache.beam.io.PubSubIO"
> > > > > > > configuration: json
> > > config, // {
> > > > "topic" :
> > > > > > "my_pubsub_topic" }
> > > > > > > }
> > > > > > >
> > > > > > >
> > > > > > > Thanks Max, this is a great
> > first step
> > > > towards defining to
> > > > > API for
> > > > > > > cross-language transforms.
> > > > > > > Is there a reason why you
> > would want
> > > to use
> > > > JSON instead
> > > > > of a proto
> > > > > > > here. I guess we'll be
> > providing a more
> > > > user friendly language
> > > > > > wrapper
> > > > > > > (for example, Python) for
> > end-users
> > > here, so
> > > > > > user-friendliness-wise, the
> > > > > > > format we choose won't matter
> > much (for
> > > > pipeline authors).
> > > > > > > If we don't support UDFs,
> > performance
> > > > difference will be
> > > > > > negligible, but
> > > > > > > UDFs might require a callback
> to
> > > original
> > > > SDK (per-element
> > > > > worst
> > > > > > case).
> > > > > > > So might make sense to choose
> > the more
> > > > efficient format.
> > > > > > >
> > > > > > > Also, probably we need to
> > define a more
> > > > expanded definition
> > > > > > (proto/JSON)
> > > > > > > to support UDFs. For example,
> a
> > > payload + a
> > > > set of parameter
> > > > > > definitions
> > > > > > > so that the target SDK (for
> > example,
> > > Java)
> > > > can call back the
> > > > > > original
> > > > > > > SDK where the pipeline was
> > authored
> > > in (for
> > > > example,
> > > > > Python) to
> > > > > > resolve
> > > > > > > UDFs at runtime.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Cham
> > > > > > >
> > > > > > > That's more generic and
> > could be
> > > used
> > > > for other languages
> > > > > > where we
> > > > > > > might
> > > > > > > have sources/sinks.
> > > > > > >
> > > > > > > > (FWIW, I was imagining
> > PubSubIO
> > > > already had a
> > > > > translation into
> > > > > > > BeamFnApi protos that
> fully
> > > specified
> > > > it, and we use
> > > > > that same
> > > > > > > format to translate back
> > out.)
> > > > > > >
> > > > > > > Not that I know of.
> > > > > > >
> > > > > > > On 01.02.19 14:02, Robert
> > > Bradshaw wrote:
> > > > > > > �� > Are you suggesting
> > something
> > > akin to
> > > > a generic
> > > > > > > >
> > > > > > > > urn:
> > > JsonConfiguredJavaSource
> > > > > > > > payload: some json
> > > specifying
> > > > which source and
> > > > > which
> > > > > > parameters
> > > > > > > >
> > > > > > > > which would expand to
> > actually
> > > > constructing and
> > > > > applying
> > > > > > that source?
> > > > > > > >
> > > > > > > > (FWIW, I was imagining
> > PubSubIO
> > > > already had a
> > > > > translation into
> > > > > > > BeamFnApi
> > > > > > > > protos that fully
> > specified
> > > it, and
> > > > we use that
> > > > > same format to
> > > > > > > translate
> > > > > > > > back out.)
> > > > > > > >
> > > > > > > > On Fri, Feb 1, 2019 at
> > 1:44 PM
> > > > Maximilian Michels
> > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > Recaping here:
> > > > > > > >
> > > > > > > > We all agree that
> > SDF is
> > > the way
> > > > to go for future
> > > > > > > implementations of
> > > > > > > > sources. It
> > enables us to get
> > > > rid of the source
> > > > > > interfaces.
> > > > > > > However,
> > > > > > > > SDF
> > > > > > > > does not solve the
> > lack of
> > > > streaming sources in
> > > > > Python.
> > > > > > > >
> > > > > > > > The expansion PR
> > (thanks
> > > btw!)
> > > > solves the
> > > > > problem of
> > > > > > > >
> expanding/translating
> > > URNs known
> > > > to an
> > > > > > ExpansionService. That
> > > > > > > is a more
> > > > > > > > programmatic way
> > of replacing
> > > > language-specific
> > > > > performs,
> > > > > > > instead of
> > > > > > > > relying on
> translators
> > > directly
> > > > in the Runner.
> > > > > > > >
> > > > > > > > What is unsolved
> > is the
> > > > configuration of
> > > > > sources from
> > > > > > a foreign
> > > > > > > > environment. In my
> > > opinion this
> > > > is the most
> > > > > pressing issue
> > > > > > > for Python
> > > > > > > > sources, because
> > what is
> > > > PubSubIO worth in
> > > > > Python if
> > > > > > you cannot
> > > > > > > > configure it?
> > > > > > > >
> > > > > > > > What about this:
> > > > > > > >
> > > > > > > > I think it is
> > worth adding a
> > > > JSON configuration
> > > > > option for
> > > > > > > all existing
> > > > > > > > Java sources. That
> > way,
> > > we could
> > > > easily
> > > > > configure them as
> > > > > > > part of the
> > > > > > > > expansion request
> > (which
> > > would
> > > > contain a JSON
> > > > > > configuration).
> > > > > > > I'll
> > > > > > > > probably fork a
> > thread to
> > > > discuss this in more
> > > > > detail,
> > > > > > but would
> > > > > > > > like to
> > > > > > > > hear your thoughts.
> > > > > > > >
> > > > > > > > -Max
> > > > > > > >
> > > > > > > > On 01.02.19 13:08,
> > Robert
> > > > Bradshaw wrote:
> > > > > > > > > On Thu, Jan 31,
> > 2019
> > > at 6:25
> > > > PM Maximilian
> > > > > Michels
> > > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > > >
> > > > > > > > > Ah, I
> > thought you
> > > meant
> > > > native Flink
> > > > > transforms.
> > > > > > > > >
> > > > > > > > > Exactly! The
> > > translation
> > > > code is already
> > > > > there.
> > > > > > The main
> > > > > > > > challenge
> > > > > > > > > is how to
> > > > > > > > >
> > programmatically
> > > > configure the BeamIO from
> > > > > > Python. I
> > > > > > > suppose
> > > > > > > > that is
> > > > > > > > > also an
> > > > > > > > > unsolved
> > problem for
> > > > cross-language
> > > > > transforms
> > > > > > in general.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > This is what
> > > > > >
> > https://github.com/apache/beam/pull/7316 does.
> > > > > > > > >
> > > > > > > > > For a particular
> > > source, one
> > > > would want to
> > > > > define a
> > > > > > URN and
> > > > > > > > > corresponding
> > payload,
> > > then
> > > > (probably) a
> > > > > > CompositeTransform in
> > > > > > > > Python
> > > > > > > > > that takes the
> > users
> > > > arguments, packages
> > > > > them into the
> > > > > > > payload,
> > > > > > > > applies
> > > > > > > > > the
> > ExternalTransform, and
> > > > returns the
> > > > > results. How
> > > > > > to handle
> > > > > > > > arbitrary
> > > > > > > > > UDFs embedded in
> > > sources is
> > > > still TBD.
> > > > > > > > >
> > > > > > > > > For
> > Matthias' pipeline
> > > > with PubSubIO we
> > > > > can build
> > > > > > > something
> > > > > > > > > specific,
> > but for
> > > > > > > > > the general
> > case there
> > > > should be way to
> > > > > > initialize a
> > > > > > > Beam IO
> > > > > > > > via a
> > > > > > > > >
> configuration
> > > > > > > > > map
> > provided by an
> > > > external environment.
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > I thought quite
> > a bit
> > > about
> > > > how we could
> > > > > represent
> > > > > > expansions
> > > > > > > > statically
> > > > > > > > > (e.g. have some
> > kind of
> > > > expansion template that
> > > > > > could be
> > > > > > > used, at
> > > > > > > > least
> > > > > > > > > in many cases,
> > as data
> > > > without firing up a
> > > > > separate
> > > > > > > process. May be
> > > > > > > > > worth doing
> > > eventually, but
> > > > we run into the
> > > > > same issues
> > > > > > > that were
> > > > > > > > > discussed at
> > > > > > > > >
> > > > > > >
> > > > >
> > > >
> https://github.com/apache/beam/pull/7316#discussion_r249996455
> > > > ).
> > > > > > > > >
> > > > > > > > > If one is
> > already using a
> > > > portable runner
> > > > > like Flink,
> > > > > > > having the job
> > > > > > > > > service process
> > > automatically
> > > > also serve up an
> > > > > > expansion
> > > > > > > service for
> > > > > > > > > various URNs it
> > knows and
> > > > cares about is
> > > > > probably a
> > > > > > pretty
> > > > > > > low bar.
> > > > > > > > > Flink could
> > serve up
> > > things
> > > > it would rather
> > > > > get back
> > > > > > > untouched in a
> > > > > > > > > transform with
> > a special
> > > > flink runner urn.
> > > > > > > > >
> > > > > > > > > As Ahmet
> > mentions, SDF is
> > > > better solution. I
> > > > > hope
> > > > > > it's not
> > > > > > > that far
> > > > > > > > > away, but even
> > once it
> > > comes
> > > > we'll likely
> > > > > want the
> > > > > > above
> > > > > > > > framework to
> > > > > > > > > invoke the full
> > suite
> > > of Java
> > > > IOs even after
> > > > > they're
> > > > > > > running on SDF
> > > > > > > > > themselves.
> > > > > > > > >
> > > > > > > > > - Robert
> > > > > > > > >
> > > > > > > > > On 31.01.19
> > 17:36,
> > > Thomas
> > > > Weise wrote:
> > > > > > > > > > Exactly,
> > that's
> > > what I
> > > > had in mind.
> > > > > > > > > >
> > > > > > > > > > A Flink
> > runner
> > > native
> > > > transform would
> > > > > make
> > > > > > the existing
> > > > > > > > unbounded
> > > > > > > > > sources
> > > > > > > > > >
> available,
> > > similar to:
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu,
> > Jan 31,
> > > 2019
> > > > at 8:18 AM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > > >
> > > > > > > > > >
> > Wouldn't it be
> > > > even more useful
> > > > > for the
> > > > > > transition
> > > > > > > > period if
> > > > > > > > > we enabled
> > Beam IO
> > > > > > > > > > to
> > be used via
> > > > Flink (like in the
> > > > > legacy
> > > > > > Flink
> > > > > > > Runner)? In
> > > > > > > > > this
> particular
> > > > > > > > > >
> example,
> > > Matthias
> > > > wants to use
> > > > > PubSubIO,
> > > > > > which
> > > > > > > is not even
> > > > > > > > > available
> as a
> > > > > > > > > >
> native
> > > > > > > > > > Flink
> > > transform.
> > > > > > > > > >
> > > > > > > > > > On
> > 31.01.19
> > > 16:21,
> > > > Thomas Weise
> > > > > wrote:
> > > > > > > > > > >
> > Until SDF is
> > > > supported, we
> > > > > could also add
> > > > > > > Flink runner
> > > > > > > > > native
> > transforms for
> > > > > > > > > > >
> > selected
> > > > unbounded sources [1].
> > > > > > > > > > >
> > > > > > > > > > >
> That
> > > might be a
> > > > reasonable
> > > > > option to
> > > > > > unblock
> > > > > > > users that
> > > > > > > > > want to try
> > Python
> > > > > > > > > > >
> > > streaming on Flink.
> > > > > > > > > > >
> > > > > > > > > > >
> > Thomas
> > > > > > > > > > >
> > > > > > > > > > >
> [1]
> > > > > > > > > > >
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > > > >
> > > > > > > > > > >
> > > > > > > > > > >
> > On Thu,
> > > Jan 31,
> > > > 2019 at 6:51 AM
> > > > > > Maximilian
> > > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > > >
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>> wrote:
> > > > > > > > > > >
> > > > > > > > > > >
> > > I
> > > have a
> > > > hard time to
> > > > > imagine
> > > > > > how can
> > > > > > > we map
> > > > > > > > in a
> > > > > > > > > generic way
> > > > > > > > > > >
> > > > RestrictionTrackers into
> > > > > the existing
> > > > > > > > >
> > > Bounded/UnboundedSource,
> > > > so I would
> > > > > > > > > > >
> > love to
> > > > hear more about
> > > > > the details.
> > > > > > > > > > >
> > > > > > > > > > >
> > Isn't it
> > > > the other way around?
> > > > > > The SDF is a
> > > > > > > > >
> > generalization of
> > > > > > > > > >
> > > UnboundedSource.
> > > > > > > > > > >
> > So
> > > we would
> > > > wrap
> > > > > UnboundedSource
> > > > > > using
> > > > > > > SDF. I'm not
> > > > > > > > > saying it is
> > > > > > > > > >
> > trivial, but
> > > > > > > > > > >
> > SDF
> > > offers
> > > > all the
> > > > > functionality that
> > > > > > > > UnboundedSource
> > > > > > > > > needs.
> > > > > > > > > > >
> > > > > > > > > > >
> > For
> > > > example, the
> > > > > > @GetInitialRestriction
> > > > > > > method
> > > > > > > > would
> > > > > > > > > call split
> > on the
> > > > > > > > > > >
> > > > UnboundedSource and the
> > > > > restriction
> > > > > > > trackers would
> > > > > > > > > then be
> used to
> > > > > > > > > >
> > process the
> > > > > > > > > > >
> > splits.
> > > > > > > > > > >
> > > > > > > > > > >
> > On
> > > 31.01.19
> > > > 15:16, Ismaël
> > > > > Mejía
> > > > > > wrote:
> > > > > > > > > > >
> > >> Not
> > > > necessarily. This
> > > > > would
> > > > > > be one
> > > > > > > way. Another
> > > > > > > > > way is
> > build an SDF
> > > > > > > > > > >
> > > wrapper for
> > > > UnboundedSource.
> > > > > > Probably the
> > > > > > > > easier path
> > > > > > > > > for
> migration.
> > > > > > > > > > >
> > >
> > > > > > > > > > >
> > > That
> > > > would be fantastic, I
> > > > > > have heard
> > > > > > > about such
> > > > > > > > > wrapper
> > multiple
> > > > > > > > > > >
> > > times
> > > > but so far there
> > > > > is not any
> > > > > > > realistic
> > > > > > > > > proposal. I
> > have a
> > > hard
> > > > > > > > > > >
> > >
> > > time to
> > > > imagine how can
> > > > > we map
> > > > > > in a
> > > > > > > generic way
> > > > > > > > >
> > RestrictionTrackers
> > > > > > > > > > >
> > >
> > > into the
> > > > existing
> > > > > > > Bounded/UnboundedSource,
> so I
> > > > > > > > > would love
> > to hear
> > > > > > > > > > >
> > > more
> > > > about the details.
> > > > > > > > > > >
> > >
> > > > > > > > > > >
> > >
> > > On Thu,
> > > > Jan 31, 2019 at
> > > > > 3:07 PM
> > > > > > > Maximilian
> > > > > > > > Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > > >
> > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>> wrote:
> > > > > > > > > > >
> > >>
> > > > > > > > > > >
> > >>
> > > > In
> > > > addition to have
> > > > > > support in
> > > > > > > the runners,
> > > > > > > > > this will
> > require a
> > > > > > > > > > >
> > >> >
> > > > rewrite of PubsubIO to
> > > > > > use the
> > > > > > > new SDF API.
> > > > > > > > > > >
> > >>
> > > > > > > > > > >
> > >> Not
> > > > necessarily. This
> > > > > would
> > > > > > be one
> > > > > > > way. Another
> > > > > > > > > way is
> > build an SDF
> > > > > > > > > > >
> > > wrapper for
> > > > > > > > > > >
> > >>
> > > > UnboundedSource.
> > > > > Probably the
> > > > > > easier
> > > > > > > path for
> > > > > > > > > migration.
> > > > > > > > > > >
> > >>
> > > > > > > > > > >
> > >> On
> > > > 31.01.19 14:03, Ismaël
> > > > > > Mejía wrote:
> > > > > > > > > > >
> > >>>>
> > > > Fortunately, there
> > > > > is already a
> > > > > > > pending
> > > > > > > > PR for
> > > > > > > > >
> cross-language
> > > > > > > > > > >
> > > pipelines which
> > > > > > > > > > >
> > > >>>> will
> > > > allow us to use
> > > > > Java
> > > > > > IO like
> > > > > > > PubSub in
> > > > > > > > > Python jobs.
> > > > > > > > > > >
> > >>>
> > > > > > > > > > >
> > >>> In
> > > > addition to have
> > > > > support
> > > > > > in the
> > > > > > > > runners, this
> > > > > > > > > will
> require a
> > > > > > > > > > >
> > >>>
> > > > rewrite of PubsubIO
> > > > > to use
> > > > > > the new
> > > > > > > SDF API.
> > > > > > > > > > >
> > >>>
> > > > > > > > > > >
> > >>> On
> > > > Thu, Jan 31, 2019
> > > > > at 12:23 PM
> > > > > > > > Maximilian Michels
> > > > > > > > > >
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > > >
> > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mail
> > > >
> > >
> >
>
Re: Cross-language transform API
Posted by Maximilian Michels <mx...@apache.org>.
Hi Cham,
Thank you for the summary of our discussion. For reference I just wanted
to mention that, in addition to the design document [1], there is a
pending PR [2]. Both need to be adapted to reflect the latest state of
the discussed design.
Let me quickly recap:
The goal of the PR is to expose the GenerateSequence transform from the
Java SDK to other SDKs. The current version of the PR uses a
transform-specific Proto message, i.e. GenerateSequencePayload [3]. The
message is constructed by the authoring SDK and then sent to the
ExpansionService which passes on the payload to a builder in
GenerateSequence.
This approach requires a) a transform-specific payload b)
transform-specific wrappers in the authoring SDK c) transform-specific
builders in the external SDK. We decided that it should be easier for
transform authors to expose their transforms to other SDKs by just
defining an interface directly in the transform. This very much goes
back to an earlier version of the discussion in this thread where we
proposed a JSON builder interface to pass parameters to transforms. We
realized that this will be sufficient for most use cases, but we wanted
the schema to be provided by the transform author.
Changes to the PR:
1) Remove GenerateSequencePayload
2) Add generic ExternalTransformPayload, for example:
message ExternalTransformPayload {
enum Enum {
PROPERTIES = 0
[(beam_urn) = "beam:external:transform:external_transform:v1"];
}
// the format may change to map<string, bytes> if types are supported
map<string, string> parameters = 1;
}
I think we haven't fully figured out the wire format, but we have agreed
that it is a list of parameters which use a predefined set of data
types. For the scope of the PR, I'll stick with strings.
3) Define a Pojo in GenerateSequence with the parameters
4) Add a schema-based Pojo <=> ExternalTransformPayload conversion
5) Let Python wrappers use ExternalTransformPayload for the request to
the ExpansionService
Once this is done, we should be able to expose any Java transform by
defining the above-mentioned Pojo. Let me know if I missed anything.
Thanks,
Max
[1] https://s.apache.org/beam-cross-language-io
[2] https://github.com/apache/beam/pull/7875
[3] message GenerateSequencePayload {
enum Enum {
PROPERTIES = 0 [(beam_urn) =
"beam:external:transform:generate_sequence:v1"];
}
int64 start = 1;
oneof stop {
bool stop_provided = 2;
int64 stop_value = 3;
}
oneof elements_per_period {
bool elements_per_period_provided = 4;
int64 elements_per_period_value = 5;
}
google.protobuf.Duration period = 6;
google.protobuf.Duration max_read_time = 7;
}
On 05.03.19 00:28, Chamikara Jayalath wrote:
> (former title was "Beam Python streaming pipeline on Flink Runner")
>
> Hi All,
>
> We have some comment threads in the doc and had a couple of offline
> meetings on the matters discussed in this thread. Here's a summary of
> the discussions so far.
>
> For a pipeline defined in SDK A to use a transform T defined in SDK B,
> we need following.
>
> (1) Extensions that will allow SDK B (or a service provided by SDK B) to
> instantiate and expand the transform with a given set of parameters.
>
> We are gravitating towards (at least as the first version) defining a
> TransformBuilder interface that will take an object P (a POJO for Java)
> as a parameter. Author of transform T will be implementing this
> interface. SDK B will inspect type of P and will generate a proto/schema
> that will serve as a language-neutral representation of it. Probably it
> should be enough to simply define a proto that is a single Map of
> strings to objects where strings define the parameters and objects
> define corresponding values that should be used for parameters. We might
> be able to re-use some of the utilities developed for Beam schema
> feature for generating a schema from a given POJO.
>
> (2) A wire format to plumb through parameters defined in SDK A to SDK B.
>
> If we use a proto Map to define the interface this should be
> straightforward. Other options are to use JSON, a more generic proto
> definition, or a blob of bytes (which is fully passed by a transform
> builder).
>
> (3) Transform stubs in SDK A so that pipeline authors can refer to
> transform T when defining the pipeline.
>
> Probably we should be able to define T in SDK A using a generic
> transform that takes a URN and a payload that is a set of key,value
> pairs (a dictionary for Python) as parameters. During transform
> expansion, this will be converted to proper wire format and will be sent
> to a transform expansion service offered by SDK B. SDK B will use this
> information to build a payload of type P, will instantiate T, will
> expand it, and will return the result of the expansion to SDK A. SDK A
> will include the expanded definition of T in the job description which
> will be submitted to a runner for execution. Author of transform T
> could manually define a wrapper API that provides a nicer
> language-specific API for T for SDK A if needed though these will have
> to be kept manually up-to date to match with interface of T in SDK B.
> In the future we can look into automatically generating nicer stubs for
> remote transforms as a a part of Beam release process.
>
> Thanks,
> Cham
>
> On Thu, Feb 14, 2019 at 8:06 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> I've revised the document and included your feedback:
> https://s.apache.org/beam-cross-language-io
>
> I think it reads much better now. I moved away from the JSON
> configuration in favor of an explicit Proto-based configuration
> approach
> which leaves it up to the transform what to include in the Proto
> payload. Transforms could still be using JSON if it is feasible.
>
> Let me know what you think.
>
> -Max
>
> On 08.02.19 16:07, Robert Bradshaw wrote:
> > On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Thank you for your comments. They will help to iterate over
> the ideas.
> >
> > > I'd like to point out that though this seems to be
> specifically
> > targeting IOs, there's nothing here that is specific to IOs.
> >
> > I thought about this from an IO perspective but I agree that
> it amounts
> > to using any type of cross-language transform. However, there
> is the IO
> > specific question of how to parameterize IO transforms.
> >
> >
> > On the other hand, I don't think paramaterizing IO transforms is
> > different than parameterizing any other kind of transform.
> >
> > If we end up
> > using PipelineOptions for that, there would really be no
> difference
> > anymore.
> >
> >
> > FWIW, I'm not suggesting that we use the pipeline-level
> PipelineOptions
> > (as this makes it impossible to configure the same transform
> differently
> > in different parts of the transform). Rather, this is an existing
> > cross-langauge configuration object/pattern we could possibly
> leverage.
> > (JSON is much easier to provide from Python (and likely other
> > languages), but it's harder to know how to read it from the Java
> side,
> > especially if it has UDFs, coders, schemas, etc. embedded.)
> >
> > @Ismael Will add a link after another review round.
> >
> > On 08.02.19 09:25, Robert Bradshaw wrote:
> > > Thanks for writing this up. I'd like to point out that though
> > this seems
> > > to be specifically targeting IOs, there's nothing here that is
> > specific
> > > to IOs.
> > >
> > > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath
> > <chamikara@google.com <ma...@google.com>
> <mailto:chamikara@google.com <ma...@google.com>>
> > > <mailto:chamikara@google.com <ma...@google.com>
> <mailto:chamikara@google.com <ma...@google.com>>>> wrote:
> > >
> > > Thanks Max. Added few comments.
> > >
> > > - Cham
> > >
> > > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía
> > <iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>
> > > <mailto:iemejia@gmail.com <ma...@gmail.com>
> <mailto:iemejia@gmail.com <ma...@gmail.com>>>> wrote:
> > >
> > > Can you please add the link to the design document
> webpage.
> > >
> > > Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> a écrit :
> > >
> > > I've created an initial design document:
> > > https://s.apache.org/beam-cross-language-io
> > >
> > > It does not contain all the details but
> perhaps it's
> > a good
> > > basis for a
> > > discussion on how we proceed.
> > >
> > > -Max
> > >
> > > On 06.02.19 19:49, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 8:38 AM Maximilian
> Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > Thanks for your replies Robert and Cham.
> > > >
> > > > What I had in mind was a generic
> Wrapper that
> > would
> > > easily allow users
> > > > to use IO from Java. Such wrapper could
> start
> > as an
> > > experimental
> > > > feature
> > > > and then, through URN versioning,
> become stable
> > > eventually.
> > > >
> > > > UDFs are needed, though they are a special
> > case. Most
> > > users (including
> > > > Matthias) just want to specify a few String
> > options
> > > which do not
> > > > require
> > > > UDFs but something along the lines what I
> > proposed here.
> > > >
> > > >
> > > > Sounds good let's start
> documenting/implementing the
> > > "easy" case and
> > > > think bit more regarding UDFs.
> > > >
> > > >
> > > >
> > > > Robert wrote:
> > > > > UDFs that are called from within an
> IO as
> > part of
> > > its operation is
> > > > > still an open question.
> > > >
> > > > Exactly. How about we solve the easier
> case first,
> > > unblock users, and
> > > > then think more about solving the
> general case?
> > > >
> > > > Cham wrote:
> > > > > I'm happy to work with you to
> realize this.
> > > >
> > > > Would be great to exchange more ideas
> on this!
> > I can
> > > compile the
> > > > current
> > > > ideas we have in a document and we move
> from
> > there.
> > > >
> > > >
> > > > Great. Thanks.
> > > >
> > > >
> > > > Thanks,
> > > > Max
> > > >
> > > > On 05.02.19 17:56, Chamikara Jayalath
> wrote:
> > > > >
> > > > >
> > > > > On Tue, Feb 5, 2019, 8:11 AM
> Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Good points Cham.
> > > > >
> > > > > JSON seemed like the most
> intuitive way to
> > > specify a
> > > > configuration map.
> > > > > We already use JSON in other places,
> > e.g. to
> > > specify the
> > > > environment
> > > > > configuration. It is not
> necessarily a
> > > contradiction to have JSON
> > > > > inside
> > > > > Protobuf. From the perspective of IO
> > authors, the
> > > > user-friendliness
> > > > > plays a role because they
> wouldn't have to
> > > deal with Protobuf.
> > > > >
> > > > >
> > > > > It's a good point that JSON will
> make this more
> > > user-friendly for IO
> > > > > transforms authors. Probably we
> should do a
> > bit of
> > > > experimentation and
> > > > > keep this experimental in case we hit a
> > > performance snag.
> > > > >
> > > > >
> > > > > I agree that the configuration
> format is an
> > > implementation
> > > > detail that
> > > > > will be hidden to users via
> easy-to-use
> > wrappers.
> > > > >
> > > > > Do we have to support UDFs for
> expanding
> > > existing IO? Users
> > > > would still
> > > > > be able to apply UDFs via ParDo
> on the IO
> > > output collections.
> > > > Generally
> > > > > speaking, I can see how for
> cross-language
> > > transforms UDF
> > > > support would
> > > > > be good. For example, a Combine
> > implementation
> > > in Java, where the
> > > > > combine UDFs come from Python.
> > > > >
> > > > >
> > > > > I think we should try to support UDFs in
> > the first
> > > version unless
> > > > > there's a major blocker that hinders
> realizing
> > > this. Many IO
> > > > transforms
> > > > > available today expect users to pass
> UDFs to
> > > realize various
> > > > features
> > > > > (for example, dynamic destinations for
> > BigqueryIO
> > > and FileIO,
> > > > timestamp
> > > > > function for KafkaIO). I think without
> > support for
> > > UDFs usability of
> > > > > cross-language transforms feature
> will be
> > > significantly limited.
> > > > >
> > > > > I'm happy to work with you to
> realize this.
> > > > >
> > > > > Thanks,
> > > > > Cham
> > > > >
> > > > >
> > > > > I suppose the question is, do we
> try to
> > solve
> > > the general
> > > > case, or
> > > > > do we
> > > > > go with a simpler approach for
> enabling the
> > > use of existing
> > > > IO first?
> > > > > Lack of IO seems to be the most
> > pressing issue
> > > for the
> > > > adoption of Beam
> > > > > Python. I imagine that a
> > backwards-compatible
> > > incremental
> > > > support for
> > > > > cross-language transforms (IOs
> first, later
> > > other transforms)
> > > > would be
> > > > > possible.
> > > > >
> > > > > -Max
> > > > >
> > > > > On 05.02.19 03:07, Chamikara
> Jayalath
> > wrote:
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 1, 2019 at 6:12 AM
> > Maximilian
> > > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > >
> > > > > > Yes, I imagine sources to
> > implement a
> > > JsonConfigurable
> > > > > interface (e.g.
> > > > > > on their builders):
> > > > > >
> > > > > > JsonConfigurable {
> > > > > > // Either a json
> string or
> > > Map<String, String>
> > > > > > apply(String jsonConfig);
> > > > > > }
> > > > > >
> > > > > > In Python we would create
> this
> > transform:
> > > > > >
> > > > > > URN: JsonConfiguredSource:v1
> > > > > > payload: {
> > > > > > environment:
> environment_id, //
> > > Java/Python/Go
> > > > > > resourceIdentifier:
> string, //
> > > > > "org.apache.beam.io.PubSubIO"
> > > > > > configuration: json
> > config, // {
> > > "topic" :
> > > > > "my_pubsub_topic" }
> > > > > > }
> > > > > >
> > > > > >
> > > > > > Thanks Max, this is a great
> first step
> > > towards defining to
> > > > API for
> > > > > > cross-language transforms.
> > > > > > Is there a reason why you
> would want
> > to use
> > > JSON instead
> > > > of a proto
> > > > > > here. I guess we'll be
> providing a more
> > > user friendly language
> > > > > wrapper
> > > > > > (for example, Python) for
> end-users
> > here, so
> > > > > user-friendliness-wise, the
> > > > > > format we choose won't matter
> much (for
> > > pipeline authors).
> > > > > > If we don't support UDFs,
> performance
> > > difference will be
> > > > > negligible, but
> > > > > > UDFs might require a callback to
> > original
> > > SDK (per-element
> > > > worst
> > > > > case).
> > > > > > So might make sense to choose
> the more
> > > efficient format.
> > > > > >
> > > > > > Also, probably we need to
> define a more
> > > expanded definition
> > > > > (proto/JSON)
> > > > > > to support UDFs. For example, a
> > payload + a
> > > set of parameter
> > > > > definitions
> > > > > > so that the target SDK (for
> example,
> > Java)
> > > can call back the
> > > > > original
> > > > > > SDK where the pipeline was
> authored
> > in (for
> > > example,
> > > > Python) to
> > > > > resolve
> > > > > > UDFs at runtime.
> > > > > >
> > > > > > Thanks,
> > > > > > Cham
> > > > > >
> > > > > > That's more generic and
> could be
> > used
> > > for other languages
> > > > > where we
> > > > > > might
> > > > > > have sources/sinks.
> > > > > >
> > > > > > > (FWIW, I was imagining
> PubSubIO
> > > already had a
> > > > translation into
> > > > > > BeamFnApi protos that fully
> > specified
> > > it, and we use
> > > > that same
> > > > > > format to translate back
> out.)
> > > > > >
> > > > > > Not that I know of.
> > > > > >
> > > > > > On 01.02.19 14:02, Robert
> > Bradshaw wrote:
> > > > > > �� > Are you suggesting
> something
> > akin to
> > > a generic
> > > > > > >
> > > > > > > urn:
> > JsonConfiguredJavaSource
> > > > > > > payload: some json
> > specifying
> > > which source and
> > > > which
> > > > > parameters
> > > > > > >
> > > > > > > which would expand to
> actually
> > > constructing and
> > > > applying
> > > > > that source?
> > > > > > >
> > > > > > > (FWIW, I was imagining
> PubSubIO
> > > already had a
> > > > translation into
> > > > > > BeamFnApi
> > > > > > > protos that fully
> specified
> > it, and
> > > we use that
> > > > same format to
> > > > > > translate
> > > > > > > back out.)
> > > > > > >
> > > > > > > On Fri, Feb 1, 2019 at
> 1:44 PM
> > > Maximilian Michels
> > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > >
> > > > > > > Recaping here:
> > > > > > >
> > > > > > > We all agree that
> SDF is
> > the way
> > > to go for future
> > > > > > implementations of
> > > > > > > sources. It
> enables us to get
> > > rid of the source
> > > > > interfaces.
> > > > > > However,
> > > > > > > SDF
> > > > > > > does not solve the
> lack of
> > > streaming sources in
> > > > Python.
> > > > > > >
> > > > > > > The expansion PR
> (thanks
> > btw!)
> > > solves the
> > > > problem of
> > > > > > > expanding/translating
> > URNs known
> > > to an
> > > > > ExpansionService. That
> > > > > > is a more
> > > > > > > programmatic way
> of replacing
> > > language-specific
> > > > performs,
> > > > > > instead of
> > > > > > > relying on translators
> > directly
> > > in the Runner.
> > > > > > >
> > > > > > > What is unsolved
> is the
> > > configuration of
> > > > sources from
> > > > > a foreign
> > > > > > > environment. In my
> > opinion this
> > > is the most
> > > > pressing issue
> > > > > > for Python
> > > > > > > sources, because
> what is
> > > PubSubIO worth in
> > > > Python if
> > > > > you cannot
> > > > > > > configure it?
> > > > > > >
> > > > > > > What about this:
> > > > > > >
> > > > > > > I think it is
> worth adding a
> > > JSON configuration
> > > > option for
> > > > > > all existing
> > > > > > > Java sources. That
> way,
> > we could
> > > easily
> > > > configure them as
> > > > > > part of the
> > > > > > > expansion request
> (which
> > would
> > > contain a JSON
> > > > > configuration).
> > > > > > I'll
> > > > > > > probably fork a
> thread to
> > > discuss this in more
> > > > detail,
> > > > > but would
> > > > > > > like to
> > > > > > > hear your thoughts.
> > > > > > >
> > > > > > > -Max
> > > > > > >
> > > > > > > On 01.02.19 13:08,
> Robert
> > > Bradshaw wrote:
> > > > > > > > On Thu, Jan 31,
> 2019
> > at 6:25
> > > PM Maximilian
> > > > Michels
> > > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > Ah, I
> thought you
> > meant
> > > native Flink
> > > > transforms.
> > > > > > > >
> > > > > > > > Exactly! The
> > translation
> > > code is already
> > > > there.
> > > > > The main
> > > > > > > challenge
> > > > > > > > is how to
> > > > > > > >
> programmatically
> > > configure the BeamIO from
> > > > > Python. I
> > > > > > suppose
> > > > > > > that is
> > > > > > > > also an
> > > > > > > > unsolved
> problem for
> > > cross-language
> > > > transforms
> > > > > in general.
> > > > > > > >
> > > > > > > >
> > > > > > > > This is what
> > > > >
> https://github.com/apache/beam/pull/7316 does.
> > > > > > > >
> > > > > > > > For a particular
> > source, one
> > > would want to
> > > > define a
> > > > > URN and
> > > > > > > > corresponding
> payload,
> > then
> > > (probably) a
> > > > > CompositeTransform in
> > > > > > > Python
> > > > > > > > that takes the
> users
> > > arguments, packages
> > > > them into the
> > > > > > payload,
> > > > > > > applies
> > > > > > > > the
> ExternalTransform, and
> > > returns the
> > > > results. How
> > > > > to handle
> > > > > > > arbitrary
> > > > > > > > UDFs embedded in
> > sources is
> > > still TBD.
> > > > > > > >
> > > > > > > > For
> Matthias' pipeline
> > > with PubSubIO we
> > > > can build
> > > > > > something
> > > > > > > > specific,
> but for
> > > > > > > > the general
> case there
> > > should be way to
> > > > > initialize a
> > > > > > Beam IO
> > > > > > > via a
> > > > > > > > configuration
> > > > > > > > map
> provided by an
> > > external environment.
> > > > > > > >
> > > > > > > >
> > > > > > > > I thought quite
> a bit
> > about
> > > how we could
> > > > represent
> > > > > expansions
> > > > > > > statically
> > > > > > > > (e.g. have some
> kind of
> > > expansion template that
> > > > > could be
> > > > > > used, at
> > > > > > > least
> > > > > > > > in many cases,
> as data
> > > without firing up a
> > > > separate
> > > > > > process. May be
> > > > > > > > worth doing
> > eventually, but
> > > we run into the
> > > > same issues
> > > > > > that were
> > > > > > > > discussed at
> > > > > > > >
> > > > > >
> > > >
> > > https://github.com/apache/beam/pull/7316#discussion_r249996455
> > > ).
> > > > > > > >
> > > > > > > > If one is
> already using a
> > > portable runner
> > > > like Flink,
> > > > > > having the job
> > > > > > > > service process
> > automatically
> > > also serve up an
> > > > > expansion
> > > > > > service for
> > > > > > > > various URNs it
> knows and
> > > cares about is
> > > > probably a
> > > > > pretty
> > > > > > low bar.
> > > > > > > > Flink could
> serve up
> > things
> > > it would rather
> > > > get back
> > > > > > untouched in a
> > > > > > > > transform with
> a special
> > > flink runner urn.
> > > > > > > >
> > > > > > > > As Ahmet
> mentions, SDF is
> > > better solution. I
> > > > hope
> > > > > it's not
> > > > > > that far
> > > > > > > > away, but even
> once it
> > comes
> > > we'll likely
> > > > want the
> > > > > above
> > > > > > > framework to
> > > > > > > > invoke the full
> suite
> > of Java
> > > IOs even after
> > > > they're
> > > > > > running on SDF
> > > > > > > > themselves.
> > > > > > > >
> > > > > > > > - Robert
> > > > > > > >
> > > > > > > > On 31.01.19
> 17:36,
> > Thomas
> > > Weise wrote:
> > > > > > > > > Exactly,
> that's
> > what I
> > > had in mind.
> > > > > > > > >
> > > > > > > > > A Flink
> runner
> > native
> > > transform would
> > > > make
> > > > > the existing
> > > > > > > unbounded
> > > > > > > > sources
> > > > > > > > > available,
> > similar to:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu,
> Jan 31,
> > 2019
> > > at 8:18 AM
> > > > Maximilian
> > > > > Michels
> > > > > > > >
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > >
> > > > > > > > >
> Wouldn't it be
> > > even more useful
> > > > for the
> > > > > transition
> > > > > > > period if
> > > > > > > > we enabled
> Beam IO
> > > > > > > > > to
> be used via
> > > Flink (like in the
> > > > legacy
> > > > > Flink
> > > > > > Runner)? In
> > > > > > > > this particular
> > > > > > > > > example,
> > Matthias
> > > wants to use
> > > > PubSubIO,
> > > > > which
> > > > > > is not even
> > > > > > > > available as a
> > > > > > > > > native
> > > > > > > > > Flink
> > transform.
> > > > > > > > >
> > > > > > > > > On
> 31.01.19
> > 16:21,
> > > Thomas Weise
> > > > wrote:
> > > > > > > > > >
> Until SDF is
> > > supported, we
> > > > could also add
> > > > > > Flink runner
> > > > > > > > native
> transforms for
> > > > > > > > > >
> selected
> > > unbounded sources [1].
> > > > > > > > > >
> > > > > > > > > > That
> > might be a
> > > reasonable
> > > > option to
> > > > > unblock
> > > > > > users that
> > > > > > > > want to try
> Python
> > > > > > > > > >
> > streaming on Flink.
> > > > > > > > > >
> > > > > > > > > >
> Thomas
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > >
> On Thu,
> > Jan 31,
> > > 2019 at 6:51 AM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > >
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>> wrote:
> > > > > > > > > >
> > > > > > > > > >
> > I
> > have a
> > > hard time to
> > > > imagine
> > > > > how can
> > > > > > we map
> > > > > > > in a
> > > > > > > > generic way
> > > > > > > > > >
> > > RestrictionTrackers into
> > > > the existing
> > > > > > > >
> > Bounded/UnboundedSource,
> > > so I would
> > > > > > > > > >
> love to
> > > hear more about
> > > > the details.
> > > > > > > > > >
> > > > > > > > > >
> Isn't it
> > > the other way around?
> > > > > The SDF is a
> > > > > > > >
> generalization of
> > > > > > > > >
> > UnboundedSource.
> > > > > > > > > >
> So
> > we would
> > > wrap
> > > > UnboundedSource
> > > > > using
> > > > > > SDF. I'm not
> > > > > > > > saying it is
> > > > > > > > >
> trivial, but
> > > > > > > > > >
> SDF
> > offers
> > > all the
> > > > functionality that
> > > > > > > UnboundedSource
> > > > > > > > needs.
> > > > > > > > > >
> > > > > > > > > >
> For
> > > example, the
> > > > > @GetInitialRestriction
> > > > > > method
> > > > > > > would
> > > > > > > > call split
> on the
> > > > > > > > > >
> > > UnboundedSource and the
> > > > restriction
> > > > > > trackers would
> > > > > > > > then be used to
> > > > > > > > >
> process the
> > > > > > > > > >
> splits.
> > > > > > > > > >
> > > > > > > > > >
> On
> > 31.01.19
> > > 15:16, Ismaël
> > > > Mejía
> > > > > wrote:
> > > > > > > > > >
> >> Not
> > > necessarily. This
> > > > would
> > > > > be one
> > > > > > way. Another
> > > > > > > > way is
> build an SDF
> > > > > > > > > >
> > wrapper for
> > > UnboundedSource.
> > > > > Probably the
> > > > > > > easier path
> > > > > > > > for migration.
> > > > > > > > > >
> >
> > > > > > > > > >
> > That
> > > would be fantastic, I
> > > > > have heard
> > > > > > about such
> > > > > > > > wrapper
> multiple
> > > > > > > > > >
> > times
> > > but so far there
> > > > is not any
> > > > > > realistic
> > > > > > > > proposal. I
> have a
> > hard
> > > > > > > > > >
> >
> > time to
> > > imagine how can
> > > > we map
> > > > > in a
> > > > > > generic way
> > > > > > > >
> RestrictionTrackers
> > > > > > > > > >
> >
> > into the
> > > existing
> > > > > > Bounded/UnboundedSource, so I
> > > > > > > > would love
> to hear
> > > > > > > > > >
> > more
> > > about the details.
> > > > > > > > > >
> >
> > > > > > > > > >
> >
> > On Thu,
> > > Jan 31, 2019 at
> > > > 3:07 PM
> > > > > > Maximilian
> > > > > > > Michels
> > > > > > > >
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>>> wrote:
> > > > > > > > > >
> >>
> > > > > > > > > >
> >>
> > > In
> > > addition to have
> > > > > support in
> > > > > > the runners,
> > > > > > > > this will
> require a
> > > > > > > > > >
> >> >
> > > rewrite of PubsubIO to
> > > > > use the
> > > > > > new SDF API.
> > > > > > > > > >
> >>
> > > > > > > > > >
> >> Not
> > > necessarily. This
> > > > would
> > > > > be one
> > > > > > way. Another
> > > > > > > > way is
> build an SDF
> > > > > > > > > >
> > wrapper for
> > > > > > > > > >
> >>
> > > UnboundedSource.
> > > > Probably the
> > > > > easier
> > > > > > path for
> > > > > > > > migration.
> > > > > > > > > >
> >>
> > > > > > > > > >
> >> On
> > > 31.01.19 14:03, Ismaël
> > > > > Mejía wrote:
> > > > > > > > > >
> >>>>
> > > Fortunately, there
> > > > is already a
> > > > > > pending
> > > > > > > PR for
> > > > > > > > cross-language
> > > > > > > > > >
> > pipelines which
> > > > > > > > > >
> > >>>> will
> > > allow us to use
> > > > Java
> > > > > IO like
> > > > > > PubSub in
> > > > > > > > Python jobs.
> > > > > > > > > >
> >>>
> > > > > > > > > >
> >>> In
> > > addition to have
> > > > support
> > > > > in the
> > > > > > > runners, this
> > > > > > > > will require a
> > > > > > > > > >
> >>>
> > > rewrite of PubsubIO
> > > > to use
> > > > > the new
> > > > > > SDF API.
> > > > > > > > > >
> >>>
> > > > > > > > > >
> >>> On
> > > Thu, Jan 31, 2019
> > > > at 12:23 PM
> > > > > > > Maximilian Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mail
> > >
> >
>
Cross-language transform API
Posted by Chamikara Jayalath <ch...@google.com>.
(former title was "Beam Python streaming pipeline on Flink Runner")
Hi All,
We have some comment threads in the doc and had a couple of offline
meetings on the matters discussed in this thread. Here's a summary of the
discussions so far.
For a pipeline defined in SDK A to use a transform T defined in SDK B, we
need following.
(1) Extensions that will allow SDK B (or a service provided by SDK B) to
instantiate and expand the transform with a given set of parameters.
We are gravitating towards (at least as the first version) defining a
TransformBuilder interface that will take an object P (a POJO for Java) as
a parameter. Author of transform T will be implementing this interface. SDK
B will inspect type of P and will generate a proto/schema that will serve
as a language-neutral representation of it. Probably it should be enough to
simply define a proto that is a single Map of strings to objects where
strings define the parameters and objects define corresponding values that
should be used for parameters. We might be able to re-use some of the
utilities developed for Beam schema feature for generating a schema from a
given POJO.
(2) A wire format to plumb through parameters defined in SDK A to SDK B.
If we use a proto Map to define the interface this should be
straightforward. Other options are to use JSON, a more generic proto
definition, or a blob of bytes (which is fully passed by a transform
builder).
(3) Transform stubs in SDK A so that pipeline authors can refer to
transform T when defining the pipeline.
Probably we should be able to define T in SDK A using a generic transform
that takes a URN and a payload that is a set of key,value pairs (a
dictionary for Python) as parameters. During transform expansion, this will
be converted to proper wire format and will be sent to a transform
expansion service offered by SDK B. SDK B will use this information to
build a payload of type P, will instantiate T, will expand it, and will
return the result of the expansion to SDK A. SDK A will include the
expanded definition of T in the job description which will be submitted to
a runner for execution. Author of transform T could manually define a
wrapper API that provides a nicer language-specific API for T for SDK A if
needed though these will have to be kept manually up-to date to match with
interface of T in SDK B. In the future we can look into automatically
generating nicer stubs for remote transforms as a a part of Beam release
process.
Thanks,
Cham
On Thu, Feb 14, 2019 at 8:06 AM Maximilian Michels <mx...@apache.org> wrote:
> I've revised the document and included your feedback:
> https://s.apache.org/beam-cross-language-io
>
> I think it reads much better now. I moved away from the JSON
> configuration in favor of an explicit Proto-based configuration approach
> which leaves it up to the transform what to include in the Proto
> payload. Transforms could still be using JSON if it is feasible.
>
> Let me know what you think.
>
> -Max
>
> On 08.02.19 16:07, Robert Bradshaw wrote:
> > On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Thank you for your comments. They will help to iterate over the
> ideas.
> >
> > > I'd like to point out that though this seems to be specifically
> > targeting IOs, there's nothing here that is specific to IOs.
> >
> > I thought about this from an IO perspective but I agree that it
> amounts
> > to using any type of cross-language transform. However, there is the
> IO
> > specific question of how to parameterize IO transforms.
> >
> >
> > On the other hand, I don't think paramaterizing IO transforms is
> > different than parameterizing any other kind of transform.
> >
> > If we end up
> > using PipelineOptions for that, there would really be no difference
> > anymore.
> >
> >
> > FWIW, I'm not suggesting that we use the pipeline-level PipelineOptions
> > (as this makes it impossible to configure the same transform differently
> > in different parts of the transform). Rather, this is an existing
> > cross-langauge configuration object/pattern we could possibly leverage.
> > (JSON is much easier to provide from Python (and likely other
> > languages), but it's harder to know how to read it from the Java side,
> > especially if it has UDFs, coders, schemas, etc. embedded.)
> >
> > @Ismael Will add a link after another review round.
> >
> > On 08.02.19 09:25, Robert Bradshaw wrote:
> > > Thanks for writing this up. I'd like to point out that though
> > this seems
> > > to be specifically targeting IOs, there's nothing here that is
> > specific
> > > to IOs.
> > >
> > > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath
> > <chamikara@google.com <ma...@google.com>
> > > <mailto:chamikara@google.com <ma...@google.com>>>
> wrote:
> > >
> > > Thanks Max. Added few comments.
> > >
> > > - Cham
> > >
> > > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía
> > <iemejia@gmail.com <ma...@gmail.com>
> > > <mailto:iemejia@gmail.com <ma...@gmail.com>>> wrote:
> > >
> > > Can you please add the link to the design document
> webpage.
> > >
> > > Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> a
> écrit :
> > >
> > > I've created an initial design document:
> > > https://s.apache.org/beam-cross-language-io
> > >
> > > It does not contain all the details but perhaps it's
> > a good
> > > basis for a
> > > discussion on how we proceed.
> > >
> > > -Max
> > >
> > > On 06.02.19 19:49, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Thanks for your replies Robert and Cham.
> > > >
> > > > What I had in mind was a generic Wrapper that
> > would
> > > easily allow users
> > > > to use IO from Java. Such wrapper could start
> > as an
> > > experimental
> > > > feature
> > > > and then, through URN versioning, become stable
> > > eventually.
> > > >
> > > > UDFs are needed, though they are a special
> > case. Most
> > > users (including
> > > > Matthias) just want to specify a few String
> > options
> > > which do not
> > > > require
> > > > UDFs but something along the lines what I
> > proposed here.
> > > >
> > > >
> > > > Sounds good let's start documenting/implementing
> the
> > > "easy" case and
> > > > think bit more regarding UDFs.
> > > >
> > > >
> > > >
> > > > Robert wrote:
> > > > > UDFs that are called from within an IO as
> > part of
> > > its operation is
> > > > > still an open question.
> > > >
> > > > Exactly. How about we solve the easier case
> first,
> > > unblock users, and
> > > > then think more about solving the general case?
> > > >
> > > > Cham wrote:
> > > > > I'm happy to work with you to realize this.
> > > >
> > > > Would be great to exchange more ideas on this!
> > I can
> > > compile the
> > > > current
> > > > ideas we have in a document and we move from
> > there.
> > > >
> > > >
> > > > Great. Thanks.
> > > >
> > > >
> > > > Thanks,
> > > > Max
> > > >
> > > > On 05.02.19 17:56, Chamikara Jayalath wrote:
> > > > >
> > > > >
> > > > > On Tue, Feb 5, 2019, 8:11 AM Maximilian
> Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > > >
> > > > > Good points Cham.
> > > > >
> > > > > JSON seemed like the most intuitive way
> to
> > > specify a
> > > > configuration map.
> > > > > We already use JSON in other places,
> > e.g. to
> > > specify the
> > > > environment
> > > > > configuration. It is not necessarily a
> > > contradiction to have JSON
> > > > > inside
> > > > > Protobuf. From the perspective of IO
> > authors, the
> > > > user-friendliness
> > > > > plays a role because they wouldn't have
> to
> > > deal with Protobuf.
> > > > >
> > > > >
> > > > > It's a good point that JSON will make this
> more
> > > user-friendly for IO
> > > > > transforms authors. Probably we should do a
> > bit of
> > > > experimentation and
> > > > > keep this experimental in case we hit a
> > > performance snag.
> > > > >
> > > > >
> > > > > I agree that the configuration format
> is an
> > > implementation
> > > > detail that
> > > > > will be hidden to users via easy-to-use
> > wrappers.
> > > > >
> > > > > Do we have to support UDFs for expanding
> > > existing IO? Users
> > > > would still
> > > > > be able to apply UDFs via ParDo on the
> IO
> > > output collections.
> > > > Generally
> > > > > speaking, I can see how for
> cross-language
> > > transforms UDF
> > > > support would
> > > > > be good. For example, a Combine
> > implementation
> > > in Java, where the
> > > > > combine UDFs come from Python.
> > > > >
> > > > >
> > > > > I think we should try to support UDFs in
> > the first
> > > version unless
> > > > > there's a major blocker that hinders
> realizing
> > > this. Many IO
> > > > transforms
> > > > > available today expect users to pass UDFs to
> > > realize various
> > > > features
> > > > > (for example, dynamic destinations for
> > BigqueryIO
> > > and FileIO,
> > > > timestamp
> > > > > function for KafkaIO). I think without
> > support for
> > > UDFs usability of
> > > > > cross-language transforms feature will be
> > > significantly limited.
> > > > >
> > > > > I'm happy to work with you to realize this.
> > > > >
> > > > > Thanks,
> > > > > Cham
> > > > >
> > > > >
> > > > > I suppose the question is, do we try to
> > solve
> > > the general
> > > > case, or
> > > > > do we
> > > > > go with a simpler approach for enabling
> the
> > > use of existing
> > > > IO first?
> > > > > Lack of IO seems to be the most
> > pressing issue
> > > for the
> > > > adoption of Beam
> > > > > Python. I imagine that a
> > backwards-compatible
> > > incremental
> > > > support for
> > > > > cross-language transforms (IOs first,
> later
> > > other transforms)
> > > > would be
> > > > > possible.
> > > > >
> > > > > -Max
> > > > >
> > > > > On 05.02.19 03:07, Chamikara Jayalath
> > wrote:
> > > > > >
> > > > > >
> > > > > > On Fri, Feb 1, 2019 at 6:12 AM
> > Maximilian
> > > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > Yes, I imagine sources to
> > implement a
> > > JsonConfigurable
> > > > > interface (e.g.
> > > > > > on their builders):
> > > > > >
> > > > > > JsonConfigurable {
> > > > > > // Either a json string or
> > > Map<String, String>
> > > > > > apply(String jsonConfig);
> > > > > > }
> > > > > >
> > > > > > In Python we would create this
> > transform:
> > > > > >
> > > > > > URN: JsonConfiguredSource:v1
> > > > > > payload: {
> > > > > > environment:
> environment_id, //
> > > Java/Python/Go
> > > > > > resourceIdentifier:
> string, //
> > > > > "org.apache.beam.io.PubSubIO"
> > > > > > configuration: json
> > config, // {
> > > "topic" :
> > > > > "my_pubsub_topic" }
> > > > > > }
> > > > > >
> > > > > >
> > > > > > Thanks Max, this is a great first
> step
> > > towards defining to
> > > > API for
> > > > > > cross-language transforms.
> > > > > > Is there a reason why you would want
> > to use
> > > JSON instead
> > > > of a proto
> > > > > > here. I guess we'll be providing a
> more
> > > user friendly language
> > > > > wrapper
> > > > > > (for example, Python) for end-users
> > here, so
> > > > > user-friendliness-wise, the
> > > > > > format we choose won't matter much
> (for
> > > pipeline authors).
> > > > > > If we don't support UDFs, performance
> > > difference will be
> > > > > negligible, but
> > > > > > UDFs might require a callback to
> > original
> > > SDK (per-element
> > > > worst
> > > > > case).
> > > > > > So might make sense to choose the
> more
> > > efficient format.
> > > > > >
> > > > > > Also, probably we need to define a
> more
> > > expanded definition
> > > > > (proto/JSON)
> > > > > > to support UDFs. For example, a
> > payload + a
> > > set of parameter
> > > > > definitions
> > > > > > so that the target SDK (for example,
> > Java)
> > > can call back the
> > > > > original
> > > > > > SDK where the pipeline was authored
> > in (for
> > > example,
> > > > Python) to
> > > > > resolve
> > > > > > UDFs at runtime.
> > > > > >
> > > > > > Thanks,
> > > > > > Cham
> > > > > >
> > > > > > That's more generic and could be
> > used
> > > for other languages
> > > > > where we
> > > > > > might
> > > > > > have sources/sinks.
> > > > > >
> > > > > > > (FWIW, I was imagining
> PubSubIO
> > > already had a
> > > > translation into
> > > > > > BeamFnApi protos that fully
> > specified
> > > it, and we use
> > > > that same
> > > > > > format to translate back out.)
> > > > > >
> > > > > > Not that I know of.
> > > > > >
> > > > > > On 01.02.19 14:02, Robert
> > Bradshaw wrote:
> > > > > > > Are you suggesting something
> > akin to
> > > a generic
> > > > > > >
> > > > > > > urn:
> > JsonConfiguredJavaSource
> > > > > > > payload: some json
> > specifying
> > > which source and
> > > > which
> > > > > parameters
> > > > > > >
> > > > > > > which would expand to actually
> > > constructing and
> > > > applying
> > > > > that source?
> > > > > > >
> > > > > > > (FWIW, I was imagining
> PubSubIO
> > > already had a
> > > > translation into
> > > > > > BeamFnApi
> > > > > > > protos that fully specified
> > it, and
> > > we use that
> > > > same format to
> > > > > > translate
> > > > > > > back out.)
> > > > > > >
> > > > > > > On Fri, Feb 1, 2019 at 1:44 PM
> > > Maximilian Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >
> > > > > > > Recaping here:
> > > > > > >
> > > > > > > We all agree that SDF is
> > the way
> > > to go for future
> > > > > > implementations of
> > > > > > > sources. It enables us to
> get
> > > rid of the source
> > > > > interfaces.
> > > > > > However,
> > > > > > > SDF
> > > > > > > does not solve the lack of
> > > streaming sources in
> > > > Python.
> > > > > > >
> > > > > > > The expansion PR (thanks
> > btw!)
> > > solves the
> > > > problem of
> > > > > > > expanding/translating
> > URNs known
> > > to an
> > > > > ExpansionService. That
> > > > > > is a more
> > > > > > > programmatic way of
> replacing
> > > language-specific
> > > > performs,
> > > > > > instead of
> > > > > > > relying on translators
> > directly
> > > in the Runner.
> > > > > > >
> > > > > > > What is unsolved is the
> > > configuration of
> > > > sources from
> > > > > a foreign
> > > > > > > environment. In my
> > opinion this
> > > is the most
> > > > pressing issue
> > > > > > for Python
> > > > > > > sources, because what is
> > > PubSubIO worth in
> > > > Python if
> > > > > you cannot
> > > > > > > configure it?
> > > > > > >
> > > > > > > What about this:
> > > > > > >
> > > > > > > I think it is worth
> adding a
> > > JSON configuration
> > > > option for
> > > > > > all existing
> > > > > > > Java sources. That way,
> > we could
> > > easily
> > > > configure them as
> > > > > > part of the
> > > > > > > expansion request (which
> > would
> > > contain a JSON
> > > > > configuration).
> > > > > > I'll
> > > > > > > probably fork a thread to
> > > discuss this in more
> > > > detail,
> > > > > but would
> > > > > > > like to
> > > > > > > hear your thoughts.
> > > > > > >
> > > > > > > -Max
> > > > > > >
> > > > > > > On 01.02.19 13:08, Robert
> > > Bradshaw wrote:
> > > > > > > > On Thu, Jan 31, 2019
> > at 6:25
> > > PM Maximilian
> > > > Michels
> > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > Ah, I thought you
> > meant
> > > native Flink
> > > > transforms.
> > > > > > > >
> > > > > > > > Exactly! The
> > translation
> > > code is already
> > > > there.
> > > > > The main
> > > > > > > challenge
> > > > > > > > is how to
> > > > > > > > programmatically
> > > configure the BeamIO from
> > > > > Python. I
> > > > > > suppose
> > > > > > > that is
> > > > > > > > also an
> > > > > > > > unsolved problem
> for
> > > cross-language
> > > > transforms
> > > > > in general.
> > > > > > > >
> > > > > > > >
> > > > > > > > This is what
> > > > > https://github.com/apache/beam/pull/7316
> does.
> > > > > > > >
> > > > > > > > For a particular
> > source, one
> > > would want to
> > > > define a
> > > > > URN and
> > > > > > > > corresponding payload,
> > then
> > > (probably) a
> > > > > CompositeTransform in
> > > > > > > Python
> > > > > > > > that takes the users
> > > arguments, packages
> > > > them into the
> > > > > > payload,
> > > > > > > applies
> > > > > > > > the ExternalTransform,
> and
> > > returns the
> > > > results. How
> > > > > to handle
> > > > > > > arbitrary
> > > > > > > > UDFs embedded in
> > sources is
> > > still TBD.
> > > > > > > >
> > > > > > > > For Matthias'
> pipeline
> > > with PubSubIO we
> > > > can build
> > > > > > something
> > > > > > > > specific, but for
> > > > > > > > the general case
> there
> > > should be way to
> > > > > initialize a
> > > > > > Beam IO
> > > > > > > via a
> > > > > > > > configuration
> > > > > > > > map provided by an
> > > external environment.
> > > > > > > >
> > > > > > > >
> > > > > > > > I thought quite a bit
> > about
> > > how we could
> > > > represent
> > > > > expansions
> > > > > > > statically
> > > > > > > > (e.g. have some kind of
> > > expansion template that
> > > > > could be
> > > > > > used, at
> > > > > > > least
> > > > > > > > in many cases, as data
> > > without firing up a
> > > > separate
> > > > > > process. May be
> > > > > > > > worth doing
> > eventually, but
> > > we run into the
> > > > same issues
> > > > > > that were
> > > > > > > > discussed at
> > > > > > > >
> > > > > >
> > > >
> > > https://github.com/apache/beam/pull/7316#discussion_r249996455
> > > ).
> > > > > > > >
> > > > > > > > If one is already
> using a
> > > portable runner
> > > > like Flink,
> > > > > > having the job
> > > > > > > > service process
> > automatically
> > > also serve up an
> > > > > expansion
> > > > > > service for
> > > > > > > > various URNs it knows
> and
> > > cares about is
> > > > probably a
> > > > > pretty
> > > > > > low bar.
> > > > > > > > Flink could serve up
> > things
> > > it would rather
> > > > get back
> > > > > > untouched in a
> > > > > > > > transform with a
> special
> > > flink runner urn.
> > > > > > > >
> > > > > > > > As Ahmet mentions, SDF
> is
> > > better solution. I
> > > > hope
> > > > > it's not
> > > > > > that far
> > > > > > > > away, but even once it
> > comes
> > > we'll likely
> > > > want the
> > > > > above
> > > > > > > framework to
> > > > > > > > invoke the full suite
> > of Java
> > > IOs even after
> > > > they're
> > > > > > running on SDF
> > > > > > > > themselves.
> > > > > > > >
> > > > > > > > - Robert
> > > > > > > >
> > > > > > > > On 31.01.19 17:36,
> > Thomas
> > > Weise wrote:
> > > > > > > > > Exactly, that's
> > what I
> > > had in mind.
> > > > > > > > >
> > > > > > > > > A Flink runner
> > native
> > > transform would
> > > > make
> > > > > the existing
> > > > > > > unbounded
> > > > > > > > sources
> > > > > > > > > available,
> > similar to:
> > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jan 31,
> > 2019
> > > at 8:18 AM
> > > > Maximilian
> > > > > Michels
> > > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > > >
> > > > > > > > > Wouldn't it
> be
> > > even more useful
> > > > for the
> > > > > transition
> > > > > > > period if
> > > > > > > > we enabled Beam IO
> > > > > > > > > to be used
> via
> > > Flink (like in the
> > > > legacy
> > > > > Flink
> > > > > > Runner)? In
> > > > > > > > this particular
> > > > > > > > > example,
> > Matthias
> > > wants to use
> > > > PubSubIO,
> > > > > which
> > > > > > is not even
> > > > > > > > available as a
> > > > > > > > > native
> > > > > > > > > Flink
> > transform.
> > > > > > > > >
> > > > > > > > > On 31.01.19
> > 16:21,
> > > Thomas Weise
> > > > wrote:
> > > > > > > > > > Until
> SDF is
> > > supported, we
> > > > could also add
> > > > > > Flink runner
> > > > > > > > native transforms
> for
> > > > > > > > > > selected
> > > unbounded sources [1].
> > > > > > > > > >
> > > > > > > > > > That
> > might be a
> > > reasonable
> > > > option to
> > > > > unblock
> > > > > > users that
> > > > > > > > want to try Python
> > > > > > > > > >
> > streaming on Flink.
> > > > > > > > > >
> > > > > > > > > > Thomas
> > > > > > > > > >
> > > > > > > > > > [1]
> > > > > > > > > >
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > > >
> > > > > > > > > >
> > > > > > > > > > On Thu,
> > Jan 31,
> > > 2019 at 6:51 AM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > > >
> > > > > > > > > > > I
> > have a
> > > hard time to
> > > > imagine
> > > > > how can
> > > > > > we map
> > > > > > > in a
> > > > > > > > generic way
> > > > > > > > > >
> > > RestrictionTrackers into
> > > > the existing
> > > > > > > >
> > Bounded/UnboundedSource,
> > > so I would
> > > > > > > > > > love
> to
> > > hear more about
> > > > the details.
> > > > > > > > > >
> > > > > > > > > >
> Isn't it
> > > the other way around?
> > > > > The SDF is a
> > > > > > > > generalization of
> > > > > > > > >
> > UnboundedSource.
> > > > > > > > > > So
> > we would
> > > wrap
> > > > UnboundedSource
> > > > > using
> > > > > > SDF. I'm not
> > > > > > > > saying it is
> > > > > > > > > trivial, but
> > > > > > > > > > SDF
> > offers
> > > all the
> > > > functionality that
> > > > > > > UnboundedSource
> > > > > > > > needs.
> > > > > > > > > >
> > > > > > > > > > For
> > > example, the
> > > > > @GetInitialRestriction
> > > > > > method
> > > > > > > would
> > > > > > > > call split on the
> > > > > > > > > >
> > > UnboundedSource and the
> > > > restriction
> > > > > > trackers would
> > > > > > > > then be used to
> > > > > > > > > process the
> > > > > > > > > >
> splits.
> > > > > > > > > >
> > > > > > > > > > On
> > 31.01.19
> > > 15:16, Ismaël
> > > > Mejía
> > > > > wrote:
> > > > > > > > > > >>
> Not
> > > necessarily. This
> > > > would
> > > > > be one
> > > > > > way. Another
> > > > > > > > way is build an SDF
> > > > > > > > > >
> > wrapper for
> > > UnboundedSource.
> > > > > Probably the
> > > > > > > easier path
> > > > > > > > for migration.
> > > > > > > > > > >
> > > > > > > > > > >
> That
> > > would be fantastic, I
> > > > > have heard
> > > > > > about such
> > > > > > > > wrapper multiple
> > > > > > > > > > >
> times
> > > but so far there
> > > > is not any
> > > > > > realistic
> > > > > > > > proposal. I have a
> > hard
> > > > > > > > > > >
> > time to
> > > imagine how can
> > > > we map
> > > > > in a
> > > > > > generic way
> > > > > > > > RestrictionTrackers
> > > > > > > > > > >
> > into the
> > > existing
> > > > > > Bounded/UnboundedSource, so I
> > > > > > > > would love to hear
> > > > > > > > > > >
> more
> > > about the details.
> > > > > > > > > > >
> > > > > > > > > > >
> > On Thu,
> > > Jan 31, 2019 at
> > > > 3:07 PM
> > > > > > Maximilian
> > > > > > > Michels
> > > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > > > >>
> > > > > > > > > > >>
> > > In
> > > addition to have
> > > > > support in
> > > > > > the runners,
> > > > > > > > this will require a
> > > > > > > > > > >>
> >
> > > rewrite of PubsubIO to
> > > > > use the
> > > > > > new SDF API.
> > > > > > > > > > >>
> > > > > > > > > > >>
> Not
> > > necessarily. This
> > > > would
> > > > > be one
> > > > > > way. Another
> > > > > > > > way is build an SDF
> > > > > > > > > >
> > wrapper for
> > > > > > > > > > >>
> > > UnboundedSource.
> > > > Probably the
> > > > > easier
> > > > > > path for
> > > > > > > > migration.
> > > > > > > > > > >>
> > > > > > > > > > >>
> On
> > > 31.01.19 14:03, Ismaël
> > > > > Mejía wrote:
> > > > > > > > > > >>>>
> > > Fortunately, there
> > > > is already a
> > > > > > pending
> > > > > > > PR for
> > > > > > > > cross-language
> > > > > > > > > >
> > pipelines which
> > > > > > > > > >
> > >>>> will
> > > allow us to use
> > > > Java
> > > > > IO like
> > > > > > PubSub in
> > > > > > > > Python jobs.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> In
> > > addition to have
> > > > support
> > > > > in the
> > > > > > > runners, this
> > > > > > > > will require a
> > > > > > > > > > >>>
> > > rewrite of PubsubIO
> > > > to use
> > > > > the new
> > > > > > SDF API.
> > > > > > > > > > >>>
> > > > > > > > > > >>>
> On
> > > Thu, Jan 31, 2019
> > > > at 12:23 PM
> > > > > > > Maximilian Michels
> > > > > > > > >
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > > >
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mail
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
I've revised the document and included your feedback:
https://s.apache.org/beam-cross-language-io
I think it reads much better now. I moved away from the JSON
configuration in favor of an explicit Proto-based configuration approach
which leaves it up to the transform what to include in the Proto
payload. Transforms could still be using JSON if it is feasible.
Let me know what you think.
-Max
On 08.02.19 16:07, Robert Bradshaw wrote:
> On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Thank you for your comments. They will help to iterate over the ideas.
>
> > I'd like to point out that though this seems to be specifically
> targeting IOs, there's nothing here that is specific to IOs.
>
> I thought about this from an IO perspective but I agree that it amounts
> to using any type of cross-language transform. However, there is the IO
> specific question of how to parameterize IO transforms.
>
>
> On the other hand, I don't think paramaterizing IO transforms is
> different than parameterizing any other kind of transform.
>
> If we end up
> using PipelineOptions for that, there would really be no difference
> anymore.
>
>
> FWIW, I'm not suggesting that we use the pipeline-level PipelineOptions
> (as this makes it impossible to configure the same transform differently
> in different parts of the transform). Rather, this is an existing
> cross-langauge configuration object/pattern we could possibly leverage.
> (JSON is much easier to provide from Python (and likely other
> languages), but it's harder to know how to read it from the Java side,
> especially if it has UDFs, coders, schemas, etc. embedded.)
>
> @Ismael Will add a link after another review round.
>
> On 08.02.19 09:25, Robert Bradshaw wrote:
> > Thanks for writing this up. I'd like to point out that though
> this seems
> > to be specifically targeting IOs, there's nothing here that is
> specific
> > to IOs.
> >
> > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath
> <chamikara@google.com <ma...@google.com>
> > <mailto:chamikara@google.com <ma...@google.com>>> wrote:
> >
> > Thanks Max. Added few comments.
> >
> > - Cham
> >
> > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía
> <iemejia@gmail.com <ma...@gmail.com>
> > <mailto:iemejia@gmail.com <ma...@gmail.com>>> wrote:
> >
> > Can you please add the link to the design document webpage.
> >
> > Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> a écrit :
> >
> > I've created an initial design document:
> > https://s.apache.org/beam-cross-language-io
> >
> > It does not contain all the details but perhaps it's
> a good
> > basis for a
> > discussion on how we proceed.
> >
> > -Max
> >
> > On 06.02.19 19:49, Chamikara Jayalath wrote:
> > >
> > >
> > > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Thanks for your replies Robert and Cham.
> > >
> > > What I had in mind was a generic Wrapper that
> would
> > easily allow users
> > > to use IO from Java. Such wrapper could start
> as an
> > experimental
> > > feature
> > > and then, through URN versioning, become stable
> > eventually.
> > >
> > > UDFs are needed, though they are a special
> case. Most
> > users (including
> > > Matthias) just want to specify a few String
> options
> > which do not
> > > require
> > > UDFs but something along the lines what I
> proposed here.
> > >
> > >
> > > Sounds good let's start documenting/implementing the
> > "easy" case and
> > > think bit more regarding UDFs.
> > >
> > >
> > >
> > > Robert wrote:
> > > > UDFs that are called from within an IO as
> part of
> > its operation is
> > > > still an open question.
> > >
> > > Exactly. How about we solve the easier case first,
> > unblock users, and
> > > then think more about solving the general case?
> > >
> > > Cham wrote:
> > > > I'm happy to work with you to realize this.
> > >
> > > Would be great to exchange more ideas on this!
> I can
> > compile the
> > > current
> > > ideas we have in a document and we move from
> there.
> > >
> > >
> > > Great. Thanks.
> > >
> > >
> > > Thanks,
> > > Max
> > >
> > > On 05.02.19 17:56, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Good points Cham.
> > > >
> > > > JSON seemed like the most intuitive way to
> > specify a
> > > configuration map.
> > > > We already use JSON in other places,
> e.g. to
> > specify the
> > > environment
> > > > configuration. It is not necessarily a
> > contradiction to have JSON
> > > > inside
> > > > Protobuf. From the perspective of IO
> authors, the
> > > user-friendliness
> > > > plays a role because they wouldn't have to
> > deal with Protobuf.
> > > >
> > > >
> > > > It's a good point that JSON will make this more
> > user-friendly for IO
> > > > transforms authors. Probably we should do a
> bit of
> > > experimentation and
> > > > keep this experimental in case we hit a
> > performance snag.
> > > >
> > > >
> > > > I agree that the configuration format is an
> > implementation
> > > detail that
> > > > will be hidden to users via easy-to-use
> wrappers.
> > > >
> > > > Do we have to support UDFs for expanding
> > existing IO? Users
> > > would still
> > > > be able to apply UDFs via ParDo on the IO
> > output collections.
> > > Generally
> > > > speaking, I can see how for cross-language
> > transforms UDF
> > > support would
> > > > be good. For example, a Combine
> implementation
> > in Java, where the
> > > > combine UDFs come from Python.
> > > >
> > > >
> > > > I think we should try to support UDFs in
> the first
> > version unless
> > > > there's a major blocker that hinders realizing
> > this. Many IO
> > > transforms
> > > > available today expect users to pass UDFs to
> > realize various
> > > features
> > > > (for example, dynamic destinations for
> BigqueryIO
> > and FileIO,
> > > timestamp
> > > > function for KafkaIO). I think without
> support for
> > UDFs usability of
> > > > cross-language transforms feature will be
> > significantly limited.
> > > >
> > > > I'm happy to work with you to realize this.
> > > >
> > > > Thanks,
> > > > Cham
> > > >
> > > >
> > > > I suppose the question is, do we try to
> solve
> > the general
> > > case, or
> > > > do we
> > > > go with a simpler approach for enabling the
> > use of existing
> > > IO first?
> > > > Lack of IO seems to be the most
> pressing issue
> > for the
> > > adoption of Beam
> > > > Python. I imagine that a
> backwards-compatible
> > incremental
> > > support for
> > > > cross-language transforms (IOs first, later
> > other transforms)
> > > would be
> > > > possible.
> > > >
> > > > -Max
> > > >
> > > > On 05.02.19 03:07, Chamikara Jayalath
> wrote:
> > > > >
> > > > >
> > > > > On Fri, Feb 1, 2019 at 6:12 AM
> Maximilian
> > Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > Yes, I imagine sources to
> implement a
> > JsonConfigurable
> > > > interface (e.g.
> > > > > on their builders):
> > > > >
> > > > > JsonConfigurable {
> > > > > // Either a json string or
> > Map<String, String>
> > > > > apply(String jsonConfig);
> > > > > }
> > > > >
> > > > > In Python we would create this
> transform:
> > > > >
> > > > > URN: JsonConfiguredSource:v1
> > > > > payload: {
> > > > > environment: environment_id, //
> > Java/Python/Go
> > > > > resourceIdentifier: string, //
> > > > "org.apache.beam.io.PubSubIO"
> > > > > configuration: json
> config, // {
> > "topic" :
> > > > "my_pubsub_topic" }
> > > > > }
> > > > >
> > > > >
> > > > > Thanks Max, this is a great first step
> > towards defining to
> > > API for
> > > > > cross-language transforms.
> > > > > Is there a reason why you would want
> to use
> > JSON instead
> > > of a proto
> > > > > here. I guess we'll be providing a more
> > user friendly language
> > > > wrapper
> > > > > (for example, Python) for end-users
> here, so
> > > > user-friendliness-wise, the
> > > > > format we choose won't matter much (for
> > pipeline authors).
> > > > > If we don't support UDFs, performance
> > difference will be
> > > > negligible, but
> > > > > UDFs might require a callback to
> original
> > SDK (per-element
> > > worst
> > > > case).
> > > > > So might make sense to choose the more
> > efficient format.
> > > > >
> > > > > Also, probably we need to define a more
> > expanded definition
> > > > (proto/JSON)
> > > > > to support UDFs. For example, a
> payload + a
> > set of parameter
> > > > definitions
> > > > > so that the target SDK (for example,
> Java)
> > can call back the
> > > > original
> > > > > SDK where the pipeline was authored
> in (for
> > example,
> > > Python) to
> > > > resolve
> > > > > UDFs at runtime.
> > > > >
> > > > > Thanks,
> > > > > Cham
> > > > >
> > > > > That's more generic and could be
> used
> > for other languages
> > > > where we
> > > > > might
> > > > > have sources/sinks.
> > > > >
> > > > > > (FWIW, I was imagining PubSubIO
> > already had a
> > > translation into
> > > > > BeamFnApi protos that fully
> specified
> > it, and we use
> > > that same
> > > > > format to translate back out.)
> > > > >
> > > > > Not that I know of.
> > > > >
> > > > > On 01.02.19 14:02, Robert
> Bradshaw wrote:
> > > > > > Are you suggesting something
> akin to
> > a generic
> > > > > >
> > > > > > urn:
> JsonConfiguredJavaSource
> > > > > > payload: some json
> specifying
> > which source and
> > > which
> > > > parameters
> > > > > >
> > > > > > which would expand to actually
> > constructing and
> > > applying
> > > > that source?
> > > > > >
> > > > > > (FWIW, I was imagining PubSubIO
> > already had a
> > > translation into
> > > > > BeamFnApi
> > > > > > protos that fully specified
> it, and
> > we use that
> > > same format to
> > > > > translate
> > > > > > back out.)
> > > > > >
> > > > > > On Fri, Feb 1, 2019 at 1:44 PM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > >
> > > > > > Recaping here:
> > > > > >
> > > > > > We all agree that SDF is
> the way
> > to go for future
> > > > > implementations of
> > > > > > sources. It enables us to get
> > rid of the source
> > > > interfaces.
> > > > > However,
> > > > > > SDF
> > > > > > does not solve the lack of
> > streaming sources in
> > > Python.
> > > > > >
> > > > > > The expansion PR (thanks
> btw!)
> > solves the
> > > problem of
> > > > > > expanding/translating
> URNs known
> > to an
> > > > ExpansionService. That
> > > > > is a more
> > > > > > programmatic way of replacing
> > language-specific
> > > performs,
> > > > > instead of
> > > > > > relying on translators
> directly
> > in the Runner.
> > > > > >
> > > > > > What is unsolved is the
> > configuration of
> > > sources from
> > > > a foreign
> > > > > > environment. In my
> opinion this
> > is the most
> > > pressing issue
> > > > > for Python
> > > > > > sources, because what is
> > PubSubIO worth in
> > > Python if
> > > > you cannot
> > > > > > configure it?
> > > > > >
> > > > > > What about this:
> > > > > >
> > > > > > I think it is worth adding a
> > JSON configuration
> > > option for
> > > > > all existing
> > > > > > Java sources. That way,
> we could
> > easily
> > > configure them as
> > > > > part of the
> > > > > > expansion request (which
> would
> > contain a JSON
> > > > configuration).
> > > > > I'll
> > > > > > probably fork a thread to
> > discuss this in more
> > > detail,
> > > > but would
> > > > > > like to
> > > > > > hear your thoughts.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > > On 01.02.19 13:08, Robert
> > Bradshaw wrote:
> > > > > > > On Thu, Jan 31, 2019
> at 6:25
> > PM Maximilian
> > > Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > >
> > > > > > > Ah, I thought you
> meant
> > native Flink
> > > transforms.
> > > > > > >
> > > > > > > Exactly! The
> translation
> > code is already
> > > there.
> > > > The main
> > > > > > challenge
> > > > > > > is how to
> > > > > > > programmatically
> > configure the BeamIO from
> > > > Python. I
> > > > > suppose
> > > > > > that is
> > > > > > > also an
> > > > > > > unsolved problem for
> > cross-language
> > > transforms
> > > > in general.
> > > > > > >
> > > > > > >
> > > > > > > This is what
> > > > https://github.com/apache/beam/pull/7316 does.
> > > > > > >
> > > > > > > For a particular
> source, one
> > would want to
> > > define a
> > > > URN and
> > > > > > > corresponding payload,
> then
> > (probably) a
> > > > CompositeTransform in
> > > > > > Python
> > > > > > > that takes the users
> > arguments, packages
> > > them into the
> > > > > payload,
> > > > > > applies
> > > > > > > the ExternalTransform, and
> > returns the
> > > results. How
> > > > to handle
> > > > > > arbitrary
> > > > > > > UDFs embedded in
> sources is
> > still TBD.
> > > > > > >
> > > > > > > For Matthias' pipeline
> > with PubSubIO we
> > > can build
> > > > > something
> > > > > > > specific, but for
> > > > > > > the general case there
> > should be way to
> > > > initialize a
> > > > > Beam IO
> > > > > > via a
> > > > > > > configuration
> > > > > > > map provided by an
> > external environment.
> > > > > > >
> > > > > > >
> > > > > > > I thought quite a bit
> about
> > how we could
> > > represent
> > > > expansions
> > > > > > statically
> > > > > > > (e.g. have some kind of
> > expansion template that
> > > > could be
> > > > > used, at
> > > > > > least
> > > > > > > in many cases, as data
> > without firing up a
> > > separate
> > > > > process. May be
> > > > > > > worth doing
> eventually, but
> > we run into the
> > > same issues
> > > > > that were
> > > > > > > discussed at
> > > > > > >
> > > > >
> > >
> > https://github.com/apache/beam/pull/7316#discussion_r249996455
> > ).
> > > > > > >
> > > > > > > If one is already using a
> > portable runner
> > > like Flink,
> > > > > having the job
> > > > > > > service process
> automatically
> > also serve up an
> > > > expansion
> > > > > service for
> > > > > > > various URNs it knows and
> > cares about is
> > > probably a
> > > > pretty
> > > > > low bar.
> > > > > > > Flink could serve up
> things
> > it would rather
> > > get back
> > > > > untouched in a
> > > > > > > transform with a special
> > flink runner urn.
> > > > > > >
> > > > > > > As Ahmet mentions, SDF is
> > better solution. I
> > > hope
> > > > it's not
> > > > > that far
> > > > > > > away, but even once it
> comes
> > we'll likely
> > > want the
> > > > above
> > > > > > framework to
> > > > > > > invoke the full suite
> of Java
> > IOs even after
> > > they're
> > > > > running on SDF
> > > > > > > themselves.
> > > > > > >
> > > > > > > - Robert
> > > > > > >
> > > > > > > On 31.01.19 17:36,
> Thomas
> > Weise wrote:
> > > > > > > > Exactly, that's
> what I
> > had in mind.
> > > > > > > >
> > > > > > > > A Flink runner
> native
> > transform would
> > > make
> > > > the existing
> > > > > > unbounded
> > > > > > > sources
> > > > > > > > available,
> similar to:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 31,
> 2019
> > at 8:18 AM
> > > Maximilian
> > > > Michels
> > > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > Wouldn't it be
> > even more useful
> > > for the
> > > > transition
> > > > > > period if
> > > > > > > we enabled Beam IO
> > > > > > > > to be used via
> > Flink (like in the
> > > legacy
> > > > Flink
> > > > > Runner)? In
> > > > > > > this particular
> > > > > > > > example,
> Matthias
> > wants to use
> > > PubSubIO,
> > > > which
> > > > > is not even
> > > > > > > available as a
> > > > > > > > native
> > > > > > > > Flink
> transform.
> > > > > > > >
> > > > > > > > On 31.01.19
> 16:21,
> > Thomas Weise
> > > wrote:
> > > > > > > > > Until SDF is
> > supported, we
> > > could also add
> > > > > Flink runner
> > > > > > > native transforms for
> > > > > > > > > selected
> > unbounded sources [1].
> > > > > > > > >
> > > > > > > > > That
> might be a
> > reasonable
> > > option to
> > > > unblock
> > > > > users that
> > > > > > > want to try Python
> > > > > > > > >
> streaming on Flink.
> > > > > > > > >
> > > > > > > > > Thomas
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu,
> Jan 31,
> > 2019 at 6:51 AM
> > > > Maximilian
> > > > > Michels
> > > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > >
> > > > > > > > > > I
> have a
> > hard time to
> > > imagine
> > > > how can
> > > > > we map
> > > > > > in a
> > > > > > > generic way
> > > > > > > > >
> > RestrictionTrackers into
> > > the existing
> > > > > > >
> Bounded/UnboundedSource,
> > so I would
> > > > > > > > > love to
> > hear more about
> > > the details.
> > > > > > > > >
> > > > > > > > > Isn't it
> > the other way around?
> > > > The SDF is a
> > > > > > > generalization of
> > > > > > > >
> UnboundedSource.
> > > > > > > > > So
> we would
> > wrap
> > > UnboundedSource
> > > > using
> > > > > SDF. I'm not
> > > > > > > saying it is
> > > > > > > > trivial, but
> > > > > > > > > SDF
> offers
> > all the
> > > functionality that
> > > > > > UnboundedSource
> > > > > > > needs.
> > > > > > > > >
> > > > > > > > > For
> > example, the
> > > > @GetInitialRestriction
> > > > > method
> > > > > > would
> > > > > > > call split on the
> > > > > > > > >
> > UnboundedSource and the
> > > restriction
> > > > > trackers would
> > > > > > > then be used to
> > > > > > > > process the
> > > > > > > > > splits.
> > > > > > > > >
> > > > > > > > > On
> 31.01.19
> > 15:16, Ismaël
> > > Mejía
> > > > wrote:
> > > > > > > > > >> Not
> > necessarily. This
> > > would
> > > > be one
> > > > > way. Another
> > > > > > > way is build an SDF
> > > > > > > > >
> wrapper for
> > UnboundedSource.
> > > > Probably the
> > > > > > easier path
> > > > > > > for migration.
> > > > > > > > > >
> > > > > > > > > > That
> > would be fantastic, I
> > > > have heard
> > > > > about such
> > > > > > > wrapper multiple
> > > > > > > > > > times
> > but so far there
> > > is not any
> > > > > realistic
> > > > > > > proposal. I have a
> hard
> > > > > > > > > >
> time to
> > imagine how can
> > > we map
> > > > in a
> > > > > generic way
> > > > > > > RestrictionTrackers
> > > > > > > > > >
> into the
> > existing
> > > > > Bounded/UnboundedSource, so I
> > > > > > > would love to hear
> > > > > > > > > > more
> > about the details.
> > > > > > > > > >
> > > > > > > > > >
> On Thu,
> > Jan 31, 2019 at
> > > 3:07 PM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>> wrote:
> > > > > > > > > >>
> > > > > > > > > >>
> > In
> > addition to have
> > > > support in
> > > > > the runners,
> > > > > > > this will require a
> > > > > > > > > >> >
> > rewrite of PubsubIO to
> > > > use the
> > > > > new SDF API.
> > > > > > > > > >>
> > > > > > > > > >> Not
> > necessarily. This
> > > would
> > > > be one
> > > > > way. Another
> > > > > > > way is build an SDF
> > > > > > > > >
> wrapper for
> > > > > > > > > >>
> > UnboundedSource.
> > > Probably the
> > > > easier
> > > > > path for
> > > > > > > migration.
> > > > > > > > > >>
> > > > > > > > > >> On
> > 31.01.19 14:03, Ismaël
> > > > Mejía wrote:
> > > > > > > > > >>>>
> > Fortunately, there
> > > is already a
> > > > > pending
> > > > > > PR for
> > > > > > > cross-language
> > > > > > > > >
> pipelines which
> > > > > > > > >
> >>>> will
> > allow us to use
> > > Java
> > > > IO like
> > > > > PubSub in
> > > > > > > Python jobs.
> > > > > > > > > >>>
> > > > > > > > > >>> In
> > addition to have
> > > support
> > > > in the
> > > > > > runners, this
> > > > > > > will require a
> > > > > > > > > >>>
> > rewrite of PubsubIO
> > > to use
> > > > the new
> > > > > SDF API.
> > > > > > > > > >>>
> > > > > > > > > >>> On
> > Thu, Jan 31, 2019
> > > at 12:23 PM
> > > > > > Maximilian Michels
> > > > > > > >
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mail
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
On Fri, Feb 8, 2019 at 3:52 PM Maximilian Michels <mx...@apache.org> wrote:
> Thank you for your comments. They will help to iterate over the ideas.
>
> > I'd like to point out that though this seems to be specifically
> targeting IOs, there's nothing here that is specific to IOs.
>
> I thought about this from an IO perspective but I agree that it amounts
> to using any type of cross-language transform. However, there is the IO
> specific question of how to parameterize IO transforms.
On the other hand, I don't think paramaterizing IO transforms is different
than parameterizing any other kind of transform.
> If we end up
> using PipelineOptions for that, there would really be no difference
> anymore.
>
FWIW, I'm not suggesting that we use the pipeline-level PipelineOptions (as
this makes it impossible to configure the same transform differently in
different parts of the transform). Rather, this is an existing
cross-langauge configuration object/pattern we could possibly leverage.
(JSON is much easier to provide from Python (and likely other languages),
but it's harder to know how to read it from the Java side, especially if it
has UDFs, coders, schemas, etc. embedded.)
> @Ismael Will add a link after another review round.
>
> On 08.02.19 09:25, Robert Bradshaw wrote:
> > Thanks for writing this up. I'd like to point out that though this seems
> > to be specifically targeting IOs, there's nothing here that is specific
> > to IOs.
> >
> > On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath <chamikara@google.com
> > <ma...@google.com>> wrote:
> >
> > Thanks Max. Added few comments.
> >
> > - Cham
> >
> > On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía <iemejia@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Can you please add the link to the design document webpage.
> >
> > Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> a écrit :
> >
> > I've created an initial design document:
> > https://s.apache.org/beam-cross-language-io
> >
> > It does not contain all the details but perhaps it's a good
> > basis for a
> > discussion on how we proceed.
> >
> > -Max
> >
> > On 06.02.19 19:49, Chamikara Jayalath wrote:
> > >
> > >
> > > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Thanks for your replies Robert and Cham.
> > >
> > > What I had in mind was a generic Wrapper that would
> > easily allow users
> > > to use IO from Java. Such wrapper could start as an
> > experimental
> > > feature
> > > and then, through URN versioning, become stable
> > eventually.
> > >
> > > UDFs are needed, though they are a special case. Most
> > users (including
> > > Matthias) just want to specify a few String options
> > which do not
> > > require
> > > UDFs but something along the lines what I proposed
> here.
> > >
> > >
> > > Sounds good let's start documenting/implementing the
> > "easy" case and
> > > think bit more regarding UDFs.
> > >
> > >
> > >
> > > Robert wrote:
> > > > UDFs that are called from within an IO as part of
> > its operation is
> > > > still an open question.
> > >
> > > Exactly. How about we solve the easier case first,
> > unblock users, and
> > > then think more about solving the general case?
> > >
> > > Cham wrote:
> > > > I'm happy to work with you to realize this.
> > >
> > > Would be great to exchange more ideas on this! I can
> > compile the
> > > current
> > > ideas we have in a document and we move from there.
> > >
> > >
> > > Great. Thanks.
> > >
> > >
> > > Thanks,
> > > Max
> > >
> > > On 05.02.19 17:56, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >
> > > > Good points Cham.
> > > >
> > > > JSON seemed like the most intuitive way to
> > specify a
> > > configuration map.
> > > > We already use JSON in other places, e.g. to
> > specify the
> > > environment
> > > > configuration. It is not necessarily a
> > contradiction to have JSON
> > > > inside
> > > > Protobuf. From the perspective of IO authors,
> the
> > > user-friendliness
> > > > plays a role because they wouldn't have to
> > deal with Protobuf.
> > > >
> > > >
> > > > It's a good point that JSON will make this more
> > user-friendly for IO
> > > > transforms authors. Probably we should do a bit of
> > > experimentation and
> > > > keep this experimental in case we hit a
> > performance snag.
> > > >
> > > >
> > > > I agree that the configuration format is an
> > implementation
> > > detail that
> > > > will be hidden to users via easy-to-use
> wrappers.
> > > >
> > > > Do we have to support UDFs for expanding
> > existing IO? Users
> > > would still
> > > > be able to apply UDFs via ParDo on the IO
> > output collections.
> > > Generally
> > > > speaking, I can see how for cross-language
> > transforms UDF
> > > support would
> > > > be good. For example, a Combine implementation
> > in Java, where the
> > > > combine UDFs come from Python.
> > > >
> > > >
> > > > I think we should try to support UDFs in the first
> > version unless
> > > > there's a major blocker that hinders realizing
> > this. Many IO
> > > transforms
> > > > available today expect users to pass UDFs to
> > realize various
> > > features
> > > > (for example, dynamic destinations for BigqueryIO
> > and FileIO,
> > > timestamp
> > > > function for KafkaIO). I think without support for
> > UDFs usability of
> > > > cross-language transforms feature will be
> > significantly limited.
> > > >
> > > > I'm happy to work with you to realize this.
> > > >
> > > > Thanks,
> > > > Cham
> > > >
> > > >
> > > > I suppose the question is, do we try to solve
> > the general
> > > case, or
> > > > do we
> > > > go with a simpler approach for enabling the
> > use of existing
> > > IO first?
> > > > Lack of IO seems to be the most pressing issue
> > for the
> > > adoption of Beam
> > > > Python. I imagine that a backwards-compatible
> > incremental
> > > support for
> > > > cross-language transforms (IOs first, later
> > other transforms)
> > > would be
> > > > possible.
> > > >
> > > > -Max
> > > >
> > > > On 05.02.19 03:07, Chamikara Jayalath wrote:
> > > > >
> > > > >
> > > > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian
> > Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Yes, I imagine sources to implement a
> > JsonConfigurable
> > > > interface (e.g.
> > > > > on their builders):
> > > > >
> > > > > JsonConfigurable {
> > > > > // Either a json string or
> > Map<String, String>
> > > > > apply(String jsonConfig);
> > > > > }
> > > > >
> > > > > In Python we would create this
> transform:
> > > > >
> > > > > URN: JsonConfiguredSource:v1
> > > > > payload: {
> > > > > environment: environment_id, //
> > Java/Python/Go
> > > > > resourceIdentifier: string, //
> > > > "org.apache.beam.io.PubSubIO"
> > > > > configuration: json config, // {
> > "topic" :
> > > > "my_pubsub_topic" }
> > > > > }
> > > > >
> > > > >
> > > > > Thanks Max, this is a great first step
> > towards defining to
> > > API for
> > > > > cross-language transforms.
> > > > > Is there a reason why you would want to use
> > JSON instead
> > > of a proto
> > > > > here. I guess we'll be providing a more
> > user friendly language
> > > > wrapper
> > > > > (for example, Python) for end-users here, so
> > > > user-friendliness-wise, the
> > > > > format we choose won't matter much (for
> > pipeline authors).
> > > > > If we don't support UDFs, performance
> > difference will be
> > > > negligible, but
> > > > > UDFs might require a callback to original
> > SDK (per-element
> > > worst
> > > > case).
> > > > > So might make sense to choose the more
> > efficient format.
> > > > >
> > > > > Also, probably we need to define a more
> > expanded definition
> > > > (proto/JSON)
> > > > > to support UDFs. For example, a payload + a
> > set of parameter
> > > > definitions
> > > > > so that the target SDK (for example, Java)
> > can call back the
> > > > original
> > > > > SDK where the pipeline was authored in (for
> > example,
> > > Python) to
> > > > resolve
> > > > > UDFs at runtime.
> > > > >
> > > > > Thanks,
> > > > > Cham
> > > > >
> > > > > That's more generic and could be used
> > for other languages
> > > > where we
> > > > > might
> > > > > have sources/sinks.
> > > > >
> > > > > > (FWIW, I was imagining PubSubIO
> > already had a
> > > translation into
> > > > > BeamFnApi protos that fully specified
> > it, and we use
> > > that same
> > > > > format to translate back out.)
> > > > >
> > > > > Not that I know of.
> > > > >
> > > > > On 01.02.19 14:02, Robert Bradshaw
> wrote:
> > > > > > Are you suggesting something akin to
> > a generic
> > > > > >
> > > > > > urn: JsonConfiguredJavaSource
> > > > > > payload: some json specifying
> > which source and
> > > which
> > > > parameters
> > > > > >
> > > > > > which would expand to actually
> > constructing and
> > > applying
> > > > that source?
> > > > > >
> > > > > > (FWIW, I was imagining PubSubIO
> > already had a
> > > translation into
> > > > > BeamFnApi
> > > > > > protos that fully specified it, and
> > we use that
> > > same format to
> > > > > translate
> > > > > > back out.)
> > > > > >
> > > > > > On Fri, Feb 1, 2019 at 1:44 PM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > Recaping here:
> > > > > >
> > > > > > We all agree that SDF is the way
> > to go for future
> > > > > implementations of
> > > > > > sources. It enables us to get
> > rid of the source
> > > > interfaces.
> > > > > However,
> > > > > > SDF
> > > > > > does not solve the lack of
> > streaming sources in
> > > Python.
> > > > > >
> > > > > > The expansion PR (thanks btw!)
> > solves the
> > > problem of
> > > > > > expanding/translating URNs known
> > to an
> > > > ExpansionService. That
> > > > > is a more
> > > > > > programmatic way of replacing
> > language-specific
> > > performs,
> > > > > instead of
> > > > > > relying on translators directly
> > in the Runner.
> > > > > >
> > > > > > What is unsolved is the
> > configuration of
> > > sources from
> > > > a foreign
> > > > > > environment. In my opinion this
> > is the most
> > > pressing issue
> > > > > for Python
> > > > > > sources, because what is
> > PubSubIO worth in
> > > Python if
> > > > you cannot
> > > > > > configure it?
> > > > > >
> > > > > > What about this:
> > > > > >
> > > > > > I think it is worth adding a
> > JSON configuration
> > > option for
> > > > > all existing
> > > > > > Java sources. That way, we could
> > easily
> > > configure them as
> > > > > part of the
> > > > > > expansion request (which would
> > contain a JSON
> > > > configuration).
> > > > > I'll
> > > > > > probably fork a thread to
> > discuss this in more
> > > detail,
> > > > but would
> > > > > > like to
> > > > > > hear your thoughts.
> > > > > >
> > > > > > -Max
> > > > > >
> > > > > > On 01.02.19 13:08, Robert
> > Bradshaw wrote:
> > > > > > > On Thu, Jan 31, 2019 at 6:25
> > PM Maximilian
> > > Michels
> > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >
> > > > > > > Ah, I thought you meant
> > native Flink
> > > transforms.
> > > > > > >
> > > > > > > Exactly! The translation
> > code is already
> > > there.
> > > > The main
> > > > > > challenge
> > > > > > > is how to
> > > > > > > programmatically
> > configure the BeamIO from
> > > > Python. I
> > > > > suppose
> > > > > > that is
> > > > > > > also an
> > > > > > > unsolved problem for
> > cross-language
> > > transforms
> > > > in general.
> > > > > > >
> > > > > > >
> > > > > > > This is what
> > > > https://github.com/apache/beam/pull/7316 does.
> > > > > > >
> > > > > > > For a particular source, one
> > would want to
> > > define a
> > > > URN and
> > > > > > > corresponding payload, then
> > (probably) a
> > > > CompositeTransform in
> > > > > > Python
> > > > > > > that takes the users
> > arguments, packages
> > > them into the
> > > > > payload,
> > > > > > applies
> > > > > > > the ExternalTransform, and
> > returns the
> > > results. How
> > > > to handle
> > > > > > arbitrary
> > > > > > > UDFs embedded in sources is
> > still TBD.
> > > > > > >
> > > > > > > For Matthias' pipeline
> > with PubSubIO we
> > > can build
> > > > > something
> > > > > > > specific, but for
> > > > > > > the general case there
> > should be way to
> > > > initialize a
> > > > > Beam IO
> > > > > > via a
> > > > > > > configuration
> > > > > > > map provided by an
> > external environment.
> > > > > > >
> > > > > > >
> > > > > > > I thought quite a bit about
> > how we could
> > > represent
> > > > expansions
> > > > > > statically
> > > > > > > (e.g. have some kind of
> > expansion template that
> > > > could be
> > > > > used, at
> > > > > > least
> > > > > > > in many cases, as data
> > without firing up a
> > > separate
> > > > > process. May be
> > > > > > > worth doing eventually, but
> > we run into the
> > > same issues
> > > > > that were
> > > > > > > discussed at
> > > > > > >
> > > > >
> > >
> >
> https://github.com/apache/beam/pull/7316#discussion_r249996455
> > ).
> > > > > > >
> > > > > > > If one is already using a
> > portable runner
> > > like Flink,
> > > > > having the job
> > > > > > > service process automatically
> > also serve up an
> > > > expansion
> > > > > service for
> > > > > > > various URNs it knows and
> > cares about is
> > > probably a
> > > > pretty
> > > > > low bar.
> > > > > > > Flink could serve up things
> > it would rather
> > > get back
> > > > > untouched in a
> > > > > > > transform with a special
> > flink runner urn.
> > > > > > >
> > > > > > > As Ahmet mentions, SDF is
> > better solution. I
> > > hope
> > > > it's not
> > > > > that far
> > > > > > > away, but even once it comes
> > we'll likely
> > > want the
> > > > above
> > > > > > framework to
> > > > > > > invoke the full suite of Java
> > IOs even after
> > > they're
> > > > > running on SDF
> > > > > > > themselves.
> > > > > > >
> > > > > > > - Robert
> > > > > > >
> > > > > > > On 31.01.19 17:36, Thomas
> > Weise wrote:
> > > > > > > > Exactly, that's what I
> > had in mind.
> > > > > > > >
> > > > > > > > A Flink runner native
> > transform would
> > > make
> > > > the existing
> > > > > > unbounded
> > > > > > > sources
> > > > > > > > available, similar to:
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 31, 2019
> > at 8:18 AM
> > > Maximilian
> > > > Michels
> > > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > Wouldn't it be
> > even more useful
> > > for the
> > > > transition
> > > > > > period if
> > > > > > > we enabled Beam IO
> > > > > > > > to be used via
> > Flink (like in the
> > > legacy
> > > > Flink
> > > > > Runner)? In
> > > > > > > this particular
> > > > > > > > example, Matthias
> > wants to use
> > > PubSubIO,
> > > > which
> > > > > is not even
> > > > > > > available as a
> > > > > > > > native
> > > > > > > > Flink transform.
> > > > > > > >
> > > > > > > > On 31.01.19 16:21,
> > Thomas Weise
> > > wrote:
> > > > > > > > > Until SDF is
> > supported, we
> > > could also add
> > > > > Flink runner
> > > > > > > native transforms for
> > > > > > > > > selected
> > unbounded sources [1].
> > > > > > > > >
> > > > > > > > > That might be a
> > reasonable
> > > option to
> > > > unblock
> > > > > users that
> > > > > > > want to try Python
> > > > > > > > > streaming on
> Flink.
> > > > > > > > >
> > > > > > > > > Thomas
> > > > > > > > >
> > > > > > > > > [1]
> > > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > > >
> > > > > > > > >
> > > > > > > > > On Thu, Jan 31,
> > 2019 at 6:51 AM
> > > > Maximilian
> > > > > Michels
> > > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> wrote:
> > > > > > > > >
> > > > > > > > > > I have a
> > hard time to
> > > imagine
> > > > how can
> > > > > we map
> > > > > > in a
> > > > > > > generic way
> > > > > > > > >
> > RestrictionTrackers into
> > > the existing
> > > > > > > Bounded/UnboundedSource,
> > so I would
> > > > > > > > > love to
> > hear more about
> > > the details.
> > > > > > > > >
> > > > > > > > > Isn't it
> > the other way around?
> > > > The SDF is a
> > > > > > > generalization of
> > > > > > > > UnboundedSource.
> > > > > > > > > So we would
> > wrap
> > > UnboundedSource
> > > > using
> > > > > SDF. I'm not
> > > > > > > saying it is
> > > > > > > > trivial, but
> > > > > > > > > SDF offers
> > all the
> > > functionality that
> > > > > > UnboundedSource
> > > > > > > needs.
> > > > > > > > >
> > > > > > > > > For
> > example, the
> > > > @GetInitialRestriction
> > > > > method
> > > > > > would
> > > > > > > call split on the
> > > > > > > > >
> > UnboundedSource and the
> > > restriction
> > > > > trackers would
> > > > > > > then be used to
> > > > > > > > process the
> > > > > > > > > splits.
> > > > > > > > >
> > > > > > > > > On 31.01.19
> > 15:16, Ismaël
> > > Mejía
> > > > wrote:
> > > > > > > > > >> Not
> > necessarily. This
> > > would
> > > > be one
> > > > > way. Another
> > > > > > > way is build an SDF
> > > > > > > > > wrapper for
> > UnboundedSource.
> > > > Probably the
> > > > > > easier path
> > > > > > > for migration.
> > > > > > > > > >
> > > > > > > > > > That
> > would be fantastic, I
> > > > have heard
> > > > > about such
> > > > > > > wrapper multiple
> > > > > > > > > > times
> > but so far there
> > > is not any
> > > > > realistic
> > > > > > > proposal. I have a hard
> > > > > > > > > > time to
> > imagine how can
> > > we map
> > > > in a
> > > > > generic way
> > > > > > > RestrictionTrackers
> > > > > > > > > > into the
> > existing
> > > > > Bounded/UnboundedSource, so I
> > > > > > > would love to hear
> > > > > > > > > > more
> > about the details.
> > > > > > > > > >
> > > > > > > > > > On Thu,
> > Jan 31, 2019 at
> > > 3:07 PM
> > > > > Maximilian
> > > > > > Michels
> > > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>>
> wrote:
> > > > > > > > > >>
> > > > > > > > > >> > In
> > addition to have
> > > > support in
> > > > > the runners,
> > > > > > > this will require a
> > > > > > > > > >> >
> > rewrite of PubsubIO to
> > > > use the
> > > > > new SDF API.
> > > > > > > > > >>
> > > > > > > > > >> Not
> > necessarily. This
> > > would
> > > > be one
> > > > > way. Another
> > > > > > > way is build an SDF
> > > > > > > > > wrapper for
> > > > > > > > > >>
> > UnboundedSource.
> > > Probably the
> > > > easier
> > > > > path for
> > > > > > > migration.
> > > > > > > > > >>
> > > > > > > > > >> On
> > 31.01.19 14:03, Ismaël
> > > > Mejía wrote:
> > > > > > > > > >>>>
> > Fortunately, there
> > > is already a
> > > > > pending
> > > > > > PR for
> > > > > > > cross-language
> > > > > > > > > pipelines
> which
> > > > > > > > > >>>> will
> > allow us to use
> > > Java
> > > > IO like
> > > > > PubSub in
> > > > > > > Python jobs.
> > > > > > > > > >>>
> > > > > > > > > >>> In
> > addition to have
> > > support
> > > > in the
> > > > > > runners, this
> > > > > > > will require a
> > > > > > > > > >>>
> > rewrite of PubsubIO
> > > to use
> > > > the new
> > > > > SDF API.
> > > > > > > > > >>>
> > > > > > > > > >>> On
> > Thu, Jan 31, 2019
> > > at 12:23 PM
> > > > > > Maximilian Michels
> > > > > > > > <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > > >
> > <mailto:mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org
> > <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mail
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Thank you for your comments. They will help to iterate over the ideas.
> I'd like to point out that though this seems to be specifically targeting IOs, there's nothing here that is specific to IOs.
I thought about this from an IO perspective but I agree that it amounts
to using any type of cross-language transform. However, there is the IO
specific question of how to parameterize IO transforms. If we end up
using PipelineOptions for that, there would really be no difference
anymore.
@Ismael Will add a link after another review round.
On 08.02.19 09:25, Robert Bradshaw wrote:
> Thanks for writing this up. I'd like to point out that though this seems
> to be specifically targeting IOs, there's nothing here that is specific
> to IOs.
>
> On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath <chamikara@google.com
> <ma...@google.com>> wrote:
>
> Thanks Max. Added few comments.
>
> - Cham
>
> On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía <iemejia@gmail.com
> <ma...@gmail.com>> wrote:
>
> Can you please add the link to the design document webpage.
>
> Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> a écrit :
>
> I've created an initial design document:
> https://s.apache.org/beam-cross-language-io
>
> It does not contain all the details but perhaps it's a good
> basis for a
> discussion on how we proceed.
>
> -Max
>
> On 06.02.19 19:49, Chamikara Jayalath wrote:
> >
> >
> > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Thanks for your replies Robert and Cham.
> >
> > What I had in mind was a generic Wrapper that would
> easily allow users
> > to use IO from Java. Such wrapper could start as an
> experimental
> > feature
> > and then, through URN versioning, become stable
> eventually.
> >
> > UDFs are needed, though they are a special case. Most
> users (including
> > Matthias) just want to specify a few String options
> which do not
> > require
> > UDFs but something along the lines what I proposed here.
> >
> >
> > Sounds good let's start documenting/implementing the
> "easy" case and
> > think bit more regarding UDFs.
> >
> >
> >
> > Robert wrote:
> > > UDFs that are called from within an IO as part of
> its operation is
> > > still an open question.
> >
> > Exactly. How about we solve the easier case first,
> unblock users, and
> > then think more about solving the general case?
> >
> > Cham wrote:
> > > I'm happy to work with you to realize this.
> >
> > Would be great to exchange more ideas on this! I can
> compile the
> > current
> > ideas we have in a document and we move from there.
> >
> >
> > Great. Thanks.
> >
> >
> > Thanks,
> > Max
> >
> > On 05.02.19 17:56, Chamikara Jayalath wrote:
> > >
> > >
> > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Good points Cham.
> > >
> > > JSON seemed like the most intuitive way to
> specify a
> > configuration map.
> > > We already use JSON in other places, e.g. to
> specify the
> > environment
> > > configuration. It is not necessarily a
> contradiction to have JSON
> > > inside
> > > Protobuf. From the perspective of IO authors, the
> > user-friendliness
> > > plays a role because they wouldn't have to
> deal with Protobuf.
> > >
> > >
> > > It's a good point that JSON will make this more
> user-friendly for IO
> > > transforms authors. Probably we should do a bit of
> > experimentation and
> > > keep this experimental in case we hit a
> performance snag.
> > >
> > >
> > > I agree that the configuration format is an
> implementation
> > detail that
> > > will be hidden to users via easy-to-use wrappers.
> > >
> > > Do we have to support UDFs for expanding
> existing IO? Users
> > would still
> > > be able to apply UDFs via ParDo on the IO
> output collections.
> > Generally
> > > speaking, I can see how for cross-language
> transforms UDF
> > support would
> > > be good. For example, a Combine implementation
> in Java, where the
> > > combine UDFs come from Python.
> > >
> > >
> > > I think we should try to support UDFs in the first
> version unless
> > > there's a major blocker that hinders realizing
> this. Many IO
> > transforms
> > > available today expect users to pass UDFs to
> realize various
> > features
> > > (for example, dynamic destinations for BigqueryIO
> and FileIO,
> > timestamp
> > > function for KafkaIO). I think without support for
> UDFs usability of
> > > cross-language transforms feature will be
> significantly limited.
> > >
> > > I'm happy to work with you to realize this.
> > >
> > > Thanks,
> > > Cham
> > >
> > >
> > > I suppose the question is, do we try to solve
> the general
> > case, or
> > > do we
> > > go with a simpler approach for enabling the
> use of existing
> > IO first?
> > > Lack of IO seems to be the most pressing issue
> for the
> > adoption of Beam
> > > Python. I imagine that a backwards-compatible
> incremental
> > support for
> > > cross-language transforms (IOs first, later
> other transforms)
> > would be
> > > possible.
> > >
> > > -Max
> > >
> > > On 05.02.19 03:07, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian
> Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > Yes, I imagine sources to implement a
> JsonConfigurable
> > > interface (e.g.
> > > > on their builders):
> > > >
> > > > JsonConfigurable {
> > > > // Either a json string or
> Map<String, String>
> > > > apply(String jsonConfig);
> > > > }
> > > >
> > > > In Python we would create this transform:
> > > >
> > > > URN: JsonConfiguredSource:v1
> > > > payload: {
> > > > environment: environment_id, //
> Java/Python/Go
> > > > resourceIdentifier: string, //
> > > "org.apache.beam.io.PubSubIO"
> > > > configuration: json config, // {
> "topic" :
> > > "my_pubsub_topic" }
> > > > }
> > > >
> > > >
> > > > Thanks Max, this is a great first step
> towards defining to
> > API for
> > > > cross-language transforms.
> > > > Is there a reason why you would want to use
> JSON instead
> > of a proto
> > > > here. I guess we'll be providing a more
> user friendly language
> > > wrapper
> > > > (for example, Python) for end-users here, so
> > > user-friendliness-wise, the
> > > > format we choose won't matter much (for
> pipeline authors).
> > > > If we don't support UDFs, performance
> difference will be
> > > negligible, but
> > > > UDFs might require a callback to original
> SDK (per-element
> > worst
> > > case).
> > > > So might make sense to choose the more
> efficient format.
> > > >
> > > > Also, probably we need to define a more
> expanded definition
> > > (proto/JSON)
> > > > to support UDFs. For example, a payload + a
> set of parameter
> > > definitions
> > > > so that the target SDK (for example, Java)
> can call back the
> > > original
> > > > SDK where the pipeline was authored in (for
> example,
> > Python) to
> > > resolve
> > > > UDFs at runtime.
> > > >
> > > > Thanks,
> > > > Cham
> > > >
> > > > That's more generic and could be used
> for other languages
> > > where we
> > > > might
> > > > have sources/sinks.
> > > >
> > > > > (FWIW, I was imagining PubSubIO
> already had a
> > translation into
> > > > BeamFnApi protos that fully specified
> it, and we use
> > that same
> > > > format to translate back out.)
> > > >
> > > > Not that I know of.
> > > >
> > > > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > > > Are you suggesting something akin to
> a generic
> > > > >
> > > > > urn: JsonConfiguredJavaSource
> > > > > payload: some json specifying
> which source and
> > which
> > > parameters
> > > > >
> > > > > which would expand to actually
> constructing and
> > applying
> > > that source?
> > > > >
> > > > > (FWIW, I was imagining PubSubIO
> already had a
> > translation into
> > > > BeamFnApi
> > > > > protos that fully specified it, and
> we use that
> > same format to
> > > > translate
> > > > > back out.)
> > > > >
> > > > > On Fri, Feb 1, 2019 at 1:44 PM
> Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > Recaping here:
> > > > >
> > > > > We all agree that SDF is the way
> to go for future
> > > > implementations of
> > > > > sources. It enables us to get
> rid of the source
> > > interfaces.
> > > > However,
> > > > > SDF
> > > > > does not solve the lack of
> streaming sources in
> > Python.
> > > > >
> > > > > The expansion PR (thanks btw!)
> solves the
> > problem of
> > > > > expanding/translating URNs known
> to an
> > > ExpansionService. That
> > > > is a more
> > > > > programmatic way of replacing
> language-specific
> > performs,
> > > > instead of
> > > > > relying on translators directly
> in the Runner.
> > > > >
> > > > > What is unsolved is the
> configuration of
> > sources from
> > > a foreign
> > > > > environment. In my opinion this
> is the most
> > pressing issue
> > > > for Python
> > > > > sources, because what is
> PubSubIO worth in
> > Python if
> > > you cannot
> > > > > configure it?
> > > > >
> > > > > What about this:
> > > > >
> > > > > I think it is worth adding a
> JSON configuration
> > option for
> > > > all existing
> > > > > Java sources. That way, we could
> easily
> > configure them as
> > > > part of the
> > > > > expansion request (which would
> contain a JSON
> > > configuration).
> > > > I'll
> > > > > probably fork a thread to
> discuss this in more
> > detail,
> > > but would
> > > > > like to
> > > > > hear your thoughts.
> > > > >
> > > > > -Max
> > > > >
> > > > > On 01.02.19 13:08, Robert
> Bradshaw wrote:
> > > > > > On Thu, Jan 31, 2019 at 6:25
> PM Maximilian
> > Michels
> > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > >
> > > > > > Ah, I thought you meant
> native Flink
> > transforms.
> > > > > >
> > > > > > Exactly! The translation
> code is already
> > there.
> > > The main
> > > > > challenge
> > > > > > is how to
> > > > > > programmatically
> configure the BeamIO from
> > > Python. I
> > > > suppose
> > > > > that is
> > > > > > also an
> > > > > > unsolved problem for
> cross-language
> > transforms
> > > in general.
> > > > > >
> > > > > >
> > > > > > This is what
> > > https://github.com/apache/beam/pull/7316 does.
> > > > > >
> > > > > > For a particular source, one
> would want to
> > define a
> > > URN and
> > > > > > corresponding payload, then
> (probably) a
> > > CompositeTransform in
> > > > > Python
> > > > > > that takes the users
> arguments, packages
> > them into the
> > > > payload,
> > > > > applies
> > > > > > the ExternalTransform, and
> returns the
> > results. How
> > > to handle
> > > > > arbitrary
> > > > > > UDFs embedded in sources is
> still TBD.
> > > > > >
> > > > > > For Matthias' pipeline
> with PubSubIO we
> > can build
> > > > something
> > > > > > specific, but for
> > > > > > the general case there
> should be way to
> > > initialize a
> > > > Beam IO
> > > > > via a
> > > > > > configuration
> > > > > > map provided by an
> external environment.
> > > > > >
> > > > > >
> > > > > > I thought quite a bit about
> how we could
> > represent
> > > expansions
> > > > > statically
> > > > > > (e.g. have some kind of
> expansion template that
> > > could be
> > > > used, at
> > > > > least
> > > > > > in many cases, as data
> without firing up a
> > separate
> > > > process. May be
> > > > > > worth doing eventually, but
> we run into the
> > same issues
> > > > that were
> > > > > > discussed at
> > > > > >
> > > >
> >
> https://github.com/apache/beam/pull/7316#discussion_r249996455
> ).
> > > > > >
> > > > > > If one is already using a
> portable runner
> > like Flink,
> > > > having the job
> > > > > > service process automatically
> also serve up an
> > > expansion
> > > > service for
> > > > > > various URNs it knows and
> cares about is
> > probably a
> > > pretty
> > > > low bar.
> > > > > > Flink could serve up things
> it would rather
> > get back
> > > > untouched in a
> > > > > > transform with a special
> flink runner urn.
> > > > > >
> > > > > > As Ahmet mentions, SDF is
> better solution. I
> > hope
> > > it's not
> > > > that far
> > > > > > away, but even once it comes
> we'll likely
> > want the
> > > above
> > > > > framework to
> > > > > > invoke the full suite of Java
> IOs even after
> > they're
> > > > running on SDF
> > > > > > themselves.
> > > > > >
> > > > > > - Robert
> > > > > >
> > > > > > On 31.01.19 17:36, Thomas
> Weise wrote:
> > > > > > > Exactly, that's what I
> had in mind.
> > > > > > >
> > > > > > > A Flink runner native
> transform would
> > make
> > > the existing
> > > > > unbounded
> > > > > > sources
> > > > > > > available, similar to:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019
> at 8:18 AM
> > Maximilian
> > > Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > >
> > > > > > > Wouldn't it be
> even more useful
> > for the
> > > transition
> > > > > period if
> > > > > > we enabled Beam IO
> > > > > > > to be used via
> Flink (like in the
> > legacy
> > > Flink
> > > > Runner)? In
> > > > > > this particular
> > > > > > > example, Matthias
> wants to use
> > PubSubIO,
> > > which
> > > > is not even
> > > > > > available as a
> > > > > > > native
> > > > > > > Flink transform.
> > > > > > >
> > > > > > > On 31.01.19 16:21,
> Thomas Weise
> > wrote:
> > > > > > > > Until SDF is
> supported, we
> > could also add
> > > > Flink runner
> > > > > > native transforms for
> > > > > > > > selected
> unbounded sources [1].
> > > > > > > >
> > > > > > > > That might be a
> reasonable
> > option to
> > > unblock
> > > > users that
> > > > > > want to try Python
> > > > > > > > streaming on Flink.
> > > > > > > >
> > > > > > > > Thomas
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 31,
> 2019 at 6:51 AM
> > > Maximilian
> > > > Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > > I have a
> hard time to
> > imagine
> > > how can
> > > > we map
> > > > > in a
> > > > > > generic way
> > > > > > > >
> RestrictionTrackers into
> > the existing
> > > > > > Bounded/UnboundedSource,
> so I would
> > > > > > > > love to
> hear more about
> > the details.
> > > > > > > >
> > > > > > > > Isn't it
> the other way around?
> > > The SDF is a
> > > > > > generalization of
> > > > > > > UnboundedSource.
> > > > > > > > So we would
> wrap
> > UnboundedSource
> > > using
> > > > SDF. I'm not
> > > > > > saying it is
> > > > > > > trivial, but
> > > > > > > > SDF offers
> all the
> > functionality that
> > > > > UnboundedSource
> > > > > > needs.
> > > > > > > >
> > > > > > > > For
> example, the
> > > @GetInitialRestriction
> > > > method
> > > > > would
> > > > > > call split on the
> > > > > > > >
> UnboundedSource and the
> > restriction
> > > > trackers would
> > > > > > then be used to
> > > > > > > process the
> > > > > > > > splits.
> > > > > > > >
> > > > > > > > On 31.01.19
> 15:16, Ismaël
> > Mejía
> > > wrote:
> > > > > > > > >> Not
> necessarily. This
> > would
> > > be one
> > > > way. Another
> > > > > > way is build an SDF
> > > > > > > > wrapper for
> UnboundedSource.
> > > Probably the
> > > > > easier path
> > > > > > for migration.
> > > > > > > > >
> > > > > > > > > That
> would be fantastic, I
> > > have heard
> > > > about such
> > > > > > wrapper multiple
> > > > > > > > > times
> but so far there
> > is not any
> > > > realistic
> > > > > > proposal. I have a hard
> > > > > > > > > time to
> imagine how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers
> > > > > > > > > into the
> existing
> > > > Bounded/UnboundedSource, so I
> > > > > > would love to hear
> > > > > > > > > more
> about the details.
> > > > > > > > >
> > > > > > > > > On Thu,
> Jan 31, 2019 at
> > 3:07 PM
> > > > Maximilian
> > > > > Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>>> wrote:
> > > > > > > > >>
> > > > > > > > >> > In
> addition to have
> > > support in
> > > > the runners,
> > > > > > this will require a
> > > > > > > > >> >
> rewrite of PubsubIO to
> > > use the
> > > > new SDF API.
> > > > > > > > >>
> > > > > > > > >> Not
> necessarily. This
> > would
> > > be one
> > > > way. Another
> > > > > > way is build an SDF
> > > > > > > > wrapper for
> > > > > > > > >>
> UnboundedSource.
> > Probably the
> > > easier
> > > > path for
> > > > > > migration.
> > > > > > > > >>
> > > > > > > > >> On
> 31.01.19 14:03, Ismaël
> > > Mejía wrote:
> > > > > > > > >>>>
> Fortunately, there
> > is already a
> > > > pending
> > > > > PR for
> > > > > > cross-language
> > > > > > > > pipelines which
> > > > > > > > >>>> will
> allow us to use
> > Java
> > > IO like
> > > > PubSub in
> > > > > > Python jobs.
> > > > > > > > >>>
> > > > > > > > >>> In
> addition to have
> > support
> > > in the
> > > > > runners, this
> > > > > > will require a
> > > > > > > > >>>
> rewrite of PubsubIO
> > to use
> > > the new
> > > > SDF API.
> > > > > > > > >>>
> > > > > > > > >>> On
> Thu, Jan 31, 2019
> > at 12:23 PM
> > > > > Maximilian Michels
> > > > > > > <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>
> > > > > > > >
> <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org
> <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mail
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
On Fri, Feb 1, 2019 at 5:42 PM Thomas Weise <th...@apache.org> wrote:
>
> On Fri, Feb 1, 2019 at 6:17 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> > Max thanks for your summary. I would like to add that we agree that
>> > the runner specific translation via URN is a temporal solution until
>> > the wrappers transforms are written, is this correct? In any case this
>> > alternative standard expansion approach deserves a discussion of their
>> > own as you mention.
>>
>> Correct. Wrapping existing Beam transforms should always be preferred
>> over Runner-specific translation because the latter is not portable.
>>
>>
> From a Python user perspective, this can still be exposed as a stub,
> without having to know about the URN.
>
Yep. In the long run, I'd expect many sources to be offered as their own
easy-to-use stubs.
> Also, isn't how we expose this is orthogonal to how it is being translated?
>
Yes.
> It may even be possible to switch the stub to SDF based translation once
> that is ready.
>
Yep. The expansion would change, but that's all an internal detail iside
the composite the user doesn't care about.
>
>
>> On 01.02.19 14:25, Ismaël Mejía wrote:
>> > Thanks for the explanation Robert it makes much more sense now. (Sorry
>> > for the confusion in the mapping I mistyped the direction SDF <->
>> > Source).
>> >
>> > Status of SDF:
>> > - Support for Dynamic Work Rebalancing is WIP.
>> > - Bounded version translation is supported by all non-portable runners
>> > in a relatively naive way.
>> > - Unbounded version translation is not supported in the non-portable
>> > runners. (Let's not forget that this case may make sense too).
>> > - Portable runners translation of SDF is WIP
>> > - There is only one IO that is written based on SDF:
>> > - HBaseIO
>> > - Some other IOs should work out of the box (those based on
>> > non-splittable DoFn):
>> > - ClickhouseIO
>> > - File-based ones: TextIO, AvroIO, ParquetIO
>> > - JdbcIO
>> > - SolrIO
>> >
>> > Max thanks for your summary. I would like to add that we agree that
>> > the runner specific translation via URN is a temporal solution until
>> > the wrappers transforms are written, is this correct? In any case this
>> > alternative standard expansion approach deserves a discussion of their
>> > own as you mention.
>> >
>> > On Fri, Feb 1, 2019 at 2:02 PM Robert Bradshaw <ro...@google.com>
>> wrote:
>> >>
>> >> Are you suggesting something akin to a generic
>> >>
>> >> urn: JsonConfiguredJavaSource
>> >> payload: some json specifying which source and which parameters
>> >>
>> >> which would expand to actually constructing and applying that source?
>> >>
>> >> (FWIW, I was imagining PubSubIO already had a translation into
>> BeamFnApi protos that fully specified it, and we use that same format to
>> translate back out.)
>> >>
>> >> On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mx...@apache.org>
>> wrote:
>> >>>
>> >>> Recaping here:
>> >>>
>> >>> We all agree that SDF is the way to go for future implementations of
>> >>> sources. It enables us to get rid of the source interfaces. However,
>> SDF
>> >>> does not solve the lack of streaming sources in Python.
>> >>>
>> >>> The expansion PR (thanks btw!) solves the problem of
>> >>> expanding/translating URNs known to an ExpansionService. That is a
>> more
>> >>> programmatic way of replacing language-specific performs, instead of
>> >>> relying on translators directly in the Runner.
>> >>>
>> >>> What is unsolved is the configuration of sources from a foreign
>> >>> environment. In my opinion this is the most pressing issue for Python
>> >>> sources, because what is PubSubIO worth in Python if you cannot
>> >>> configure it?
>> >>>
>> >>> What about this:
>> >>>
>> >>> I think it is worth adding a JSON configuration option for all
>> existing
>> >>> Java sources. That way, we could easily configure them as part of the
>> >>> expansion request (which would contain a JSON configuration). I'll
>> >>> probably fork a thread to discuss this in more detail, but would like
>> to
>> >>> hear your thoughts.
>> >>>
>> >>> -Max
>> >>>
>> >>> On 01.02.19 13:08, Robert Bradshaw wrote:
>> >>>> On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
>> >>>> <ma...@apache.org>> wrote:
>> >>>>
>> >>>> Ah, I thought you meant native Flink transforms.
>> >>>>
>> >>>> Exactly! The translation code is already there. The main
>> challenge
>> >>>> is how to
>> >>>> programmatically configure the BeamIO from Python. I suppose
>> that is
>> >>>> also an
>> >>>> unsolved problem for cross-language transforms in general.
>> >>>>
>> >>>>
>> >>>> This is what https://github.com/apache/beam/pull/7316 does.
>> >>>>
>> >>>> For a particular source, one would want to define a URN and
>> >>>> corresponding payload, then (probably) a CompositeTransform in Python
>> >>>> that takes the users arguments, packages them into the payload,
>> applies
>> >>>> the ExternalTransform, and returns the results. How to handle
>> arbitrary
>> >>>> UDFs embedded in sources is still TBD.
>> >>>>
>> >>>> For Matthias' pipeline with PubSubIO we can build something
>> >>>> specific, but for
>> >>>> the general case there should be way to initialize a Beam IO
>> via a
>> >>>> configuration
>> >>>> map provided by an external environment.
>> >>>>
>> >>>>
>> >>>> I thought quite a bit about how we could represent expansions
>> statically
>> >>>> (e.g. have some kind of expansion template that could be used, at
>> least
>> >>>> in many cases, as data without firing up a separate process. May be
>> >>>> worth doing eventually, but we run into the same issues that were
>> >>>> discussed at
>> >>>> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>> >>>>
>> >>>> If one is already using a portable runner like Flink, having the job
>> >>>> service process automatically also serve up an expansion service for
>> >>>> various URNs it knows and cares about is probably a pretty low bar.
>> >>>> Flink could serve up things it would rather get back untouched in a
>> >>>> transform with a special flink runner urn.
>> >>>>
>> >>>> As Ahmet mentions, SDF is better solution. I hope it's not that far
>> >>>> away, but even once it comes we'll likely want the above framework to
>> >>>> invoke the full suite of Java IOs even after they're running on SDF
>> >>>> themselves.
>> >>>>
>> >>>> - Robert
>> >>>>
>> >>>> On 31.01.19 17:36, Thomas Weise wrote:
>> >>>> > Exactly, that's what I had in mind.
>> >>>> >
>> >>>> > A Flink runner native transform would make the existing
>> unbounded
>> >>>> sources
>> >>>> > available, similar to:
>> >>>> >
>> >>>> >
>> >>>>
>> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
>> >>>> <mxm@apache.org <ma...@apache.org>
>> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>> >>>> >
>> >>>> > Wouldn't it be even more useful for the transition
>> period if
>> >>>> we enabled Beam IO
>> >>>> > to be used via Flink (like in the legacy Flink Runner)?
>> In
>> >>>> this particular
>> >>>> > example, Matthias wants to use PubSubIO, which is not
>> even
>> >>>> available as a
>> >>>> > native
>> >>>> > Flink transform.
>> >>>> >
>> >>>> > On 31.01.19 16:21, Thomas Weise wrote:
>> >>>> > > Until SDF is supported, we could also add Flink runner
>> >>>> native transforms for
>> >>>> > > selected unbounded sources [1].
>> >>>> > >
>> >>>> > > That might be a reasonable option to unblock users
>> that
>> >>>> want to try Python
>> >>>> > > streaming on Flink.
>> >>>> > >
>> >>>> > > Thomas
>> >>>> > >
>> >>>> > > [1]
>> >>>> > >
>> >>>> >
>> >>>>
>> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>> >>>> > >
>> >>>> > >
>> >>>> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
>> >>>> <mxm@apache.org <ma...@apache.org>
>> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> >>>> > >
>> >>>> > > > I have a hard time to imagine how can we map
>> in a
>> >>>> generic way
>> >>>> > > RestrictionTrackers into the existing
>> >>>> Bounded/UnboundedSource, so I would
>> >>>> > > love to hear more about the details.
>> >>>> > >
>> >>>> > > Isn't it the other way around? The SDF is a
>> >>>> generalization of
>> >>>> > UnboundedSource.
>> >>>> > > So we would wrap UnboundedSource using SDF. I'm
>> not
>> >>>> saying it is
>> >>>> > trivial, but
>> >>>> > > SDF offers all the functionality that
>> UnboundedSource
>> >>>> needs.
>> >>>> > >
>> >>>> > > For example, the @GetInitialRestriction method
>> would
>> >>>> call split on the
>> >>>> > > UnboundedSource and the restriction trackers would
>> >>>> then be used to
>> >>>> > process the
>> >>>> > > splits.
>> >>>> > >
>> >>>> > > On 31.01.19 15:16, Ismaël Mejía wrote:
>> >>>> > > >> Not necessarily. This would be one way.
>> Another
>> >>>> way is build an SDF
>> >>>> > > wrapper for UnboundedSource. Probably the easier
>> path
>> >>>> for migration.
>> >>>> > > >
>> >>>> > > > That would be fantastic, I have heard about
>> such
>> >>>> wrapper multiple
>> >>>> > > > times but so far there is not any realistic
>> >>>> proposal. I have a hard
>> >>>> > > > time to imagine how can we map in a generic way
>> >>>> RestrictionTrackers
>> >>>> > > > into the existing Bounded/UnboundedSource, so I
>> >>>> would love to hear
>> >>>> > > > more about the details.
>> >>>> > > >
>> >>>> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian
>> Michels
>> >>>> <mxm@apache.org <ma...@apache.org>
>> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> >>>> > > >>
>> >>>> > > >> > In addition to have support in the
>> runners,
>> >>>> this will require a
>> >>>> > > >> > rewrite of PubsubIO to use the new SDF
>> API.
>> >>>> > > >>
>> >>>> > > >> Not necessarily. This would be one way.
>> Another
>> >>>> way is build an SDF
>> >>>> > > wrapper for
>> >>>> > > >> UnboundedSource. Probably the easier path for
>> >>>> migration.
>> >>>> > > >>
>> >>>> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
>> >>>> > > >>>> Fortunately, there is already a pending PR
>> for
>> >>>> cross-language
>> >>>> > > pipelines which
>> >>>> > > >>>> will allow us to use Java IO like PubSub in
>> >>>> Python jobs.
>> >>>> > > >>>
>> >>>> > > >>> In addition to have support in the runners,
>> this
>> >>>> will require a
>> >>>> > > >>> rewrite of PubsubIO to use the new SDF API.
>> >>>> > > >>>
>> >>>> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian
>> Michels
>> >>>> > <mxm@apache.org <ma...@apache.org>
>> >>>> <mailto:mxm@apache.org <ma...@apache.org>>
>> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> >>>> > > >>>>
>> >>>> > > >>>> Hi Matthias,
>> >>>> > > >>>>
>> >>>> > > >>>> This is already reflected in the
>> compatibility
>> >>>> matrix, if you look
>> >>>> > > under SDF.
>> >>>> > > >>>> There is no UnboundedSource interface for
>> >>>> portable pipelines.
>> >>>> > That's a
>> >>>> > > legacy
>> >>>> > > >>>> abstraction that will be replaced with SDF.
>> >>>> > > >>>>
>> >>>> > > >>>> Fortunately, there is already a pending PR
>> for
>> >>>> cross-language
>> >>>> > > pipelines which
>> >>>> > > >>>> will allow us to use Java IO like PubSub in
>> >>>> Python jobs.
>> >>>> > > >>>>
>> >>>> > > >>>> Thanks,
>> >>>> > > >>>> Max
>> >>>> > > >>>>
>> >>>> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
>> >>>> > > >>>>> Hey Ankur,
>> >>>> > > >>>>>
>> >>>> > > >>>>> Thanks for the swift reply. Should I change
>> >>>> this in the
>> >>>> > capability matrix
>> >>>> > > >>>>>
>> >>>> <https://s.apache.org/apache-beam-portability-support-table>
>> then?
>> >>>> > > >>>>>
>> >>>> > > >>>>> Many thanks.
>> >>>> > > >>>>> Best,
>> >>>> > > >>>>> Matthias
>> >>>> > > >>>>>
>> >>>> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
>> >>>> <goenka@google.com <ma...@google.com>
>> >>>> > <mailto:goenka@google.com <ma...@google.com>>
>> >>>> > > <mailto:goenka@google.com <mailto:
>> goenka@google.com>
>> >>>> <mailto:goenka@google.com <ma...@google.com>>>
>> >>>> > > >>>>> <mailto:goenka@google.com
>> >>>> <ma...@google.com> <mailto:goenka@google.com
>> >>>> <ma...@google.com>>
>> >>>> > <mailto:goenka@google.com <ma...@google.com>
>> >>>> <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
>> >>>> > > >>>>>
>> >>>> > > >>>>> Hi Matthias,
>> >>>> > > >>>>>
>> >>>> > > >>>>> Unfortunately, unbounded reads
>> including
>> >>>> pubsub are not yet
>> >>>> > > supported for
>> >>>> > > >>>>> portable runners.
>> >>>> > > >>>>>
>> >>>> > > >>>>> Thanks,
>> >>>> > > >>>>> Ankur
>> >>>> > > >>>>>
>> >>>> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
>> Matthias
>> >>>> Baetens
>> >>>> > > <baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com> <mailto:
>> baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>>
>> >>>> > <mailto:baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com> <mailto:
>> baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>>>
>> >>>> > > >>>>> <mailto:baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>
>> >>>> > <mailto:baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>>
>> >>>> > > <mailto:baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>
>> >>>> > <mailto:baetensmatthias@gmail.com
>> >>>> <ma...@gmail.com>>>>> wrote:
>> >>>> > > >>>>>
>> >>>> > > >>>>> Hi everyone,
>> >>>> > > >>>>>
>> >>>> > > >>>>> Last few days I have been trying
>> to
>> >>>> run a streaming
>> >>>> > > pipeline (code on
>> >>>> > > >>>>> Github
>> >>>> <https://github.com/matthiasa4/beam-demo>) on a
>> >>>> > > Flink Runner.
>> >>>> > > >>>>>
>> >>>> > > >>>>> I am running a Flink cluster
>> locally
>> >>>> (v1.5.6
>> >>>> > > >>>>>
>> >>>> <https://flink.apache.org/downloads.html>)
>> >>>> > > >>>>> I have built the SDK Harness
>> >>>> Container: /./gradlew
>> >>>> > > >>>>>
>> :beam-sdks-python-container:docker/
>> >>>> > > >>>>> and started the JobServer:
>> /./gradlew
>> >>>> > > >>>>>
>> >>>> :beam-runners-flink_2.11-job-server:runShadow
>> >>>> > > >>>>> -PflinkMasterUrl=localhost:8081./
>> >>>> > > >>>>>
>> >>>> > > >>>>> I run my pipeline with:
>> /env/bin/python
>> >>>> > streaming_pipeline.py
>> >>>> > > >>>>> --runner=PortableRunner
>> >>>> --job_endpoint=localhost:8099
>> >>>> > > --output xxx
>> >>>> > > >>>>> --input_subscription xxx
>> >>>> --output_subscription xxx/
>> >>>> > > >>>>> /
>> >>>> > > >>>>> /
>> >>>> > > >>>>> All this is running inside a
>> Ubuntu
>> >>>> (Bionic) in a
>> >>>> > Virtualbox.
>> >>>> > > >>>>>
>> >>>> > > >>>>> The job submits fine, but
>> >>>> unfortunately fails after
>> >>>> > a few
>> >>>> > > seconds with
>> >>>> > > >>>>> the error attached.
>> >>>> > > >>>>>
>> >>>> > > >>>>> Anything I am missing or doing
>> wrong?
>> >>>> > > >>>>>
>> >>>> > > >>>>> Many thanks.
>> >>>> > > >>>>> Best,
>> >>>> > > >>>>> Matthias
>> >>>> > > >>>>>
>> >>>> > > >>>>>
>> >>>> > >
>> >>>> >
>> >>>>
>>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Thomas Weise <th...@apache.org>.
On Fri, Feb 1, 2019 at 6:17 AM Maximilian Michels <mx...@apache.org> wrote:
> > Max thanks for your summary. I would like to add that we agree that
> > the runner specific translation via URN is a temporal solution until
> > the wrappers transforms are written, is this correct? In any case this
> > alternative standard expansion approach deserves a discussion of their
> > own as you mention.
>
> Correct. Wrapping existing Beam transforms should always be preferred
> over Runner-specific translation because the latter is not portable.
>
>
From a Python user perspective, this can still be exposed as a stub,
without having to know about the URN.
Also, isn't how we expose this is orthogonal to how it is being translated?
It may even be possible to switch the stub to SDF based translation once
that is ready.
> On 01.02.19 14:25, Ismaël Mejía wrote:
> > Thanks for the explanation Robert it makes much more sense now. (Sorry
> > for the confusion in the mapping I mistyped the direction SDF <->
> > Source).
> >
> > Status of SDF:
> > - Support for Dynamic Work Rebalancing is WIP.
> > - Bounded version translation is supported by all non-portable runners
> > in a relatively naive way.
> > - Unbounded version translation is not supported in the non-portable
> > runners. (Let's not forget that this case may make sense too).
> > - Portable runners translation of SDF is WIP
> > - There is only one IO that is written based on SDF:
> > - HBaseIO
> > - Some other IOs should work out of the box (those based on
> > non-splittable DoFn):
> > - ClickhouseIO
> > - File-based ones: TextIO, AvroIO, ParquetIO
> > - JdbcIO
> > - SolrIO
> >
> > Max thanks for your summary. I would like to add that we agree that
> > the runner specific translation via URN is a temporal solution until
> > the wrappers transforms are written, is this correct? In any case this
> > alternative standard expansion approach deserves a discussion of their
> > own as you mention.
> >
> > On Fri, Feb 1, 2019 at 2:02 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>
> >> Are you suggesting something akin to a generic
> >>
> >> urn: JsonConfiguredJavaSource
> >> payload: some json specifying which source and which parameters
> >>
> >> which would expand to actually constructing and applying that source?
> >>
> >> (FWIW, I was imagining PubSubIO already had a translation into
> BeamFnApi protos that fully specified it, and we use that same format to
> translate back out.)
> >>
> >> On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>>
> >>> Recaping here:
> >>>
> >>> We all agree that SDF is the way to go for future implementations of
> >>> sources. It enables us to get rid of the source interfaces. However,
> SDF
> >>> does not solve the lack of streaming sources in Python.
> >>>
> >>> The expansion PR (thanks btw!) solves the problem of
> >>> expanding/translating URNs known to an ExpansionService. That is a more
> >>> programmatic way of replacing language-specific performs, instead of
> >>> relying on translators directly in the Runner.
> >>>
> >>> What is unsolved is the configuration of sources from a foreign
> >>> environment. In my opinion this is the most pressing issue for Python
> >>> sources, because what is PubSubIO worth in Python if you cannot
> >>> configure it?
> >>>
> >>> What about this:
> >>>
> >>> I think it is worth adding a JSON configuration option for all existing
> >>> Java sources. That way, we could easily configure them as part of the
> >>> expansion request (which would contain a JSON configuration). I'll
> >>> probably fork a thread to discuss this in more detail, but would like
> to
> >>> hear your thoughts.
> >>>
> >>> -Max
> >>>
> >>> On 01.02.19 13:08, Robert Bradshaw wrote:
> >>>> On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
> >>>> <ma...@apache.org>> wrote:
> >>>>
> >>>> Ah, I thought you meant native Flink transforms.
> >>>>
> >>>> Exactly! The translation code is already there. The main
> challenge
> >>>> is how to
> >>>> programmatically configure the BeamIO from Python. I suppose
> that is
> >>>> also an
> >>>> unsolved problem for cross-language transforms in general.
> >>>>
> >>>>
> >>>> This is what https://github.com/apache/beam/pull/7316 does.
> >>>>
> >>>> For a particular source, one would want to define a URN and
> >>>> corresponding payload, then (probably) a CompositeTransform in Python
> >>>> that takes the users arguments, packages them into the payload,
> applies
> >>>> the ExternalTransform, and returns the results. How to handle
> arbitrary
> >>>> UDFs embedded in sources is still TBD.
> >>>>
> >>>> For Matthias' pipeline with PubSubIO we can build something
> >>>> specific, but for
> >>>> the general case there should be way to initialize a Beam IO via
> a
> >>>> configuration
> >>>> map provided by an external environment.
> >>>>
> >>>>
> >>>> I thought quite a bit about how we could represent expansions
> statically
> >>>> (e.g. have some kind of expansion template that could be used, at
> least
> >>>> in many cases, as data without firing up a separate process. May be
> >>>> worth doing eventually, but we run into the same issues that were
> >>>> discussed at
> >>>> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> >>>>
> >>>> If one is already using a portable runner like Flink, having the job
> >>>> service process automatically also serve up an expansion service for
> >>>> various URNs it knows and cares about is probably a pretty low bar.
> >>>> Flink could serve up things it would rather get back untouched in a
> >>>> transform with a special flink runner urn.
> >>>>
> >>>> As Ahmet mentions, SDF is better solution. I hope it's not that far
> >>>> away, but even once it comes we'll likely want the above framework to
> >>>> invoke the full suite of Java IOs even after they're running on SDF
> >>>> themselves.
> >>>>
> >>>> - Robert
> >>>>
> >>>> On 31.01.19 17:36, Thomas Weise wrote:
> >>>> > Exactly, that's what I had in mind.
> >>>> >
> >>>> > A Flink runner native transform would make the existing
> unbounded
> >>>> sources
> >>>> > available, similar to:
> >>>> >
> >>>> >
> >>>>
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> >>>> >
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> >>>> <mxm@apache.org <ma...@apache.org>
> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>>> >
> >>>> > Wouldn't it be even more useful for the transition period
> if
> >>>> we enabled Beam IO
> >>>> > to be used via Flink (like in the legacy Flink Runner)? In
> >>>> this particular
> >>>> > example, Matthias wants to use PubSubIO, which is not even
> >>>> available as a
> >>>> > native
> >>>> > Flink transform.
> >>>> >
> >>>> > On 31.01.19 16:21, Thomas Weise wrote:
> >>>> > > Until SDF is supported, we could also add Flink runner
> >>>> native transforms for
> >>>> > > selected unbounded sources [1].
> >>>> > >
> >>>> > > That might be a reasonable option to unblock users that
> >>>> want to try Python
> >>>> > > streaming on Flink.
> >>>> > >
> >>>> > > Thomas
> >>>> > >
> >>>> > > [1]
> >>>> > >
> >>>> >
> >>>>
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> >>>> > >
> >>>> > >
> >>>> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> >>>> <mxm@apache.org <ma...@apache.org>
> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>
> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> >>>> > >
> >>>> > > > I have a hard time to imagine how can we map in
> a
> >>>> generic way
> >>>> > > RestrictionTrackers into the existing
> >>>> Bounded/UnboundedSource, so I would
> >>>> > > love to hear more about the details.
> >>>> > >
> >>>> > > Isn't it the other way around? The SDF is a
> >>>> generalization of
> >>>> > UnboundedSource.
> >>>> > > So we would wrap UnboundedSource using SDF. I'm not
> >>>> saying it is
> >>>> > trivial, but
> >>>> > > SDF offers all the functionality that
> UnboundedSource
> >>>> needs.
> >>>> > >
> >>>> > > For example, the @GetInitialRestriction method
> would
> >>>> call split on the
> >>>> > > UnboundedSource and the restriction trackers would
> >>>> then be used to
> >>>> > process the
> >>>> > > splits.
> >>>> > >
> >>>> > > On 31.01.19 15:16, Ismaël Mejía wrote:
> >>>> > > >> Not necessarily. This would be one way. Another
> >>>> way is build an SDF
> >>>> > > wrapper for UnboundedSource. Probably the easier
> path
> >>>> for migration.
> >>>> > > >
> >>>> > > > That would be fantastic, I have heard about such
> >>>> wrapper multiple
> >>>> > > > times but so far there is not any realistic
> >>>> proposal. I have a hard
> >>>> > > > time to imagine how can we map in a generic way
> >>>> RestrictionTrackers
> >>>> > > > into the existing Bounded/UnboundedSource, so I
> >>>> would love to hear
> >>>> > > > more about the details.
> >>>> > > >
> >>>> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian
> Michels
> >>>> <mxm@apache.org <ma...@apache.org>
> >>>> > <mailto:mxm@apache.org <ma...@apache.org>>
> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> >>>> > > >>
> >>>> > > >> > In addition to have support in the runners,
> >>>> this will require a
> >>>> > > >> > rewrite of PubsubIO to use the new SDF API.
> >>>> > > >>
> >>>> > > >> Not necessarily. This would be one way. Another
> >>>> way is build an SDF
> >>>> > > wrapper for
> >>>> > > >> UnboundedSource. Probably the easier path for
> >>>> migration.
> >>>> > > >>
> >>>> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> >>>> > > >>>> Fortunately, there is already a pending PR
> for
> >>>> cross-language
> >>>> > > pipelines which
> >>>> > > >>>> will allow us to use Java IO like PubSub in
> >>>> Python jobs.
> >>>> > > >>>
> >>>> > > >>> In addition to have support in the runners,
> this
> >>>> will require a
> >>>> > > >>> rewrite of PubsubIO to use the new SDF API.
> >>>> > > >>>
> >>>> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian
> Michels
> >>>> > <mxm@apache.org <ma...@apache.org>
> >>>> <mailto:mxm@apache.org <ma...@apache.org>>
> >>>> > > <mailto:mxm@apache.org <ma...@apache.org>
> >>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> >>>> > > >>>>
> >>>> > > >>>> Hi Matthias,
> >>>> > > >>>>
> >>>> > > >>>> This is already reflected in the
> compatibility
> >>>> matrix, if you look
> >>>> > > under SDF.
> >>>> > > >>>> There is no UnboundedSource interface for
> >>>> portable pipelines.
> >>>> > That's a
> >>>> > > legacy
> >>>> > > >>>> abstraction that will be replaced with SDF.
> >>>> > > >>>>
> >>>> > > >>>> Fortunately, there is already a pending PR
> for
> >>>> cross-language
> >>>> > > pipelines which
> >>>> > > >>>> will allow us to use Java IO like PubSub in
> >>>> Python jobs.
> >>>> > > >>>>
> >>>> > > >>>> Thanks,
> >>>> > > >>>> Max
> >>>> > > >>>>
> >>>> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> >>>> > > >>>>> Hey Ankur,
> >>>> > > >>>>>
> >>>> > > >>>>> Thanks for the swift reply. Should I change
> >>>> this in the
> >>>> > capability matrix
> >>>> > > >>>>>
> >>>> <https://s.apache.org/apache-beam-portability-support-table>
> then?
> >>>> > > >>>>>
> >>>> > > >>>>> Many thanks.
> >>>> > > >>>>> Best,
> >>>> > > >>>>> Matthias
> >>>> > > >>>>>
> >>>> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
> >>>> <goenka@google.com <ma...@google.com>
> >>>> > <mailto:goenka@google.com <ma...@google.com>>
> >>>> > > <mailto:goenka@google.com <mailto:
> goenka@google.com>
> >>>> <mailto:goenka@google.com <ma...@google.com>>>
> >>>> > > >>>>> <mailto:goenka@google.com
> >>>> <ma...@google.com> <mailto:goenka@google.com
> >>>> <ma...@google.com>>
> >>>> > <mailto:goenka@google.com <ma...@google.com>
> >>>> <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
> >>>> > > >>>>>
> >>>> > > >>>>> Hi Matthias,
> >>>> > > >>>>>
> >>>> > > >>>>> Unfortunately, unbounded reads
> including
> >>>> pubsub are not yet
> >>>> > > supported for
> >>>> > > >>>>> portable runners.
> >>>> > > >>>>>
> >>>> > > >>>>> Thanks,
> >>>> > > >>>>> Ankur
> >>>> > > >>>>>
> >>>> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> Matthias
> >>>> Baetens
> >>>> > > <baetensmatthias@gmail.com
> >>>> <ma...@gmail.com> <mailto:
> baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>>
> >>>> > <mailto:baetensmatthias@gmail.com
> >>>> <ma...@gmail.com> <mailto:
> baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>>>
> >>>> > > >>>>> <mailto:baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>
> >>>> > <mailto:baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>>
> >>>> > > <mailto:baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>
> >>>> > <mailto:baetensmatthias@gmail.com
> >>>> <ma...@gmail.com>>>>> wrote:
> >>>> > > >>>>>
> >>>> > > >>>>> Hi everyone,
> >>>> > > >>>>>
> >>>> > > >>>>> Last few days I have been trying
> to
> >>>> run a streaming
> >>>> > > pipeline (code on
> >>>> > > >>>>> Github
> >>>> <https://github.com/matthiasa4/beam-demo>) on a
> >>>> > > Flink Runner.
> >>>> > > >>>>>
> >>>> > > >>>>> I am running a Flink cluster
> locally
> >>>> (v1.5.6
> >>>> > > >>>>>
> >>>> <https://flink.apache.org/downloads.html>)
> >>>> > > >>>>> I have built the SDK Harness
> >>>> Container: /./gradlew
> >>>> > > >>>>>
> :beam-sdks-python-container:docker/
> >>>> > > >>>>> and started the JobServer:
> /./gradlew
> >>>> > > >>>>>
> >>>> :beam-runners-flink_2.11-job-server:runShadow
> >>>> > > >>>>> -PflinkMasterUrl=localhost:8081./
> >>>> > > >>>>>
> >>>> > > >>>>> I run my pipeline with:
> /env/bin/python
> >>>> > streaming_pipeline.py
> >>>> > > >>>>> --runner=PortableRunner
> >>>> --job_endpoint=localhost:8099
> >>>> > > --output xxx
> >>>> > > >>>>> --input_subscription xxx
> >>>> --output_subscription xxx/
> >>>> > > >>>>> /
> >>>> > > >>>>> /
> >>>> > > >>>>> All this is running inside a
> Ubuntu
> >>>> (Bionic) in a
> >>>> > Virtualbox.
> >>>> > > >>>>>
> >>>> > > >>>>> The job submits fine, but
> >>>> unfortunately fails after
> >>>> > a few
> >>>> > > seconds with
> >>>> > > >>>>> the error attached.
> >>>> > > >>>>>
> >>>> > > >>>>> Anything I am missing or doing
> wrong?
> >>>> > > >>>>>
> >>>> > > >>>>> Many thanks.
> >>>> > > >>>>> Best,
> >>>> > > >>>>> Matthias
> >>>> > > >>>>>
> >>>> > > >>>>>
> >>>> > >
> >>>> >
> >>>>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
> Max thanks for your summary. I would like to add that we agree that
> the runner specific translation via URN is a temporal solution until
> the wrappers transforms are written, is this correct? In any case this
> alternative standard expansion approach deserves a discussion of their
> own as you mention.
Correct. Wrapping existing Beam transforms should always be preferred
over Runner-specific translation because the latter is not portable.
On 01.02.19 14:25, Ismaël Mejía wrote:
> Thanks for the explanation Robert it makes much more sense now. (Sorry
> for the confusion in the mapping I mistyped the direction SDF <->
> Source).
>
> Status of SDF:
> - Support for Dynamic Work Rebalancing is WIP.
> - Bounded version translation is supported by all non-portable runners
> in a relatively naive way.
> - Unbounded version translation is not supported in the non-portable
> runners. (Let's not forget that this case may make sense too).
> - Portable runners translation of SDF is WIP
> - There is only one IO that is written based on SDF:
> - HBaseIO
> - Some other IOs should work out of the box (those based on
> non-splittable DoFn):
> - ClickhouseIO
> - File-based ones: TextIO, AvroIO, ParquetIO
> - JdbcIO
> - SolrIO
>
> Max thanks for your summary. I would like to add that we agree that
> the runner specific translation via URN is a temporal solution until
> the wrappers transforms are written, is this correct? In any case this
> alternative standard expansion approach deserves a discussion of their
> own as you mention.
>
> On Fri, Feb 1, 2019 at 2:02 PM Robert Bradshaw <ro...@google.com> wrote:
>>
>> Are you suggesting something akin to a generic
>>
>> urn: JsonConfiguredJavaSource
>> payload: some json specifying which source and which parameters
>>
>> which would expand to actually constructing and applying that source?
>>
>> (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi protos that fully specified it, and we use that same format to translate back out.)
>>
>> On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mx...@apache.org> wrote:
>>>
>>> Recaping here:
>>>
>>> We all agree that SDF is the way to go for future implementations of
>>> sources. It enables us to get rid of the source interfaces. However, SDF
>>> does not solve the lack of streaming sources in Python.
>>>
>>> The expansion PR (thanks btw!) solves the problem of
>>> expanding/translating URNs known to an ExpansionService. That is a more
>>> programmatic way of replacing language-specific performs, instead of
>>> relying on translators directly in the Runner.
>>>
>>> What is unsolved is the configuration of sources from a foreign
>>> environment. In my opinion this is the most pressing issue for Python
>>> sources, because what is PubSubIO worth in Python if you cannot
>>> configure it?
>>>
>>> What about this:
>>>
>>> I think it is worth adding a JSON configuration option for all existing
>>> Java sources. That way, we could easily configure them as part of the
>>> expansion request (which would contain a JSON configuration). I'll
>>> probably fork a thread to discuss this in more detail, but would like to
>>> hear your thoughts.
>>>
>>> -Max
>>>
>>> On 01.02.19 13:08, Robert Bradshaw wrote:
>>>> On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
>>>> <ma...@apache.org>> wrote:
>>>>
>>>> Ah, I thought you meant native Flink transforms.
>>>>
>>>> Exactly! The translation code is already there. The main challenge
>>>> is how to
>>>> programmatically configure the BeamIO from Python. I suppose that is
>>>> also an
>>>> unsolved problem for cross-language transforms in general.
>>>>
>>>>
>>>> This is what https://github.com/apache/beam/pull/7316 does.
>>>>
>>>> For a particular source, one would want to define a URN and
>>>> corresponding payload, then (probably) a CompositeTransform in Python
>>>> that takes the users arguments, packages them into the payload, applies
>>>> the ExternalTransform, and returns the results. How to handle arbitrary
>>>> UDFs embedded in sources is still TBD.
>>>>
>>>> For Matthias' pipeline with PubSubIO we can build something
>>>> specific, but for
>>>> the general case there should be way to initialize a Beam IO via a
>>>> configuration
>>>> map provided by an external environment.
>>>>
>>>>
>>>> I thought quite a bit about how we could represent expansions statically
>>>> (e.g. have some kind of expansion template that could be used, at least
>>>> in many cases, as data without firing up a separate process. May be
>>>> worth doing eventually, but we run into the same issues that were
>>>> discussed at
>>>> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>>>>
>>>> If one is already using a portable runner like Flink, having the job
>>>> service process automatically also serve up an expansion service for
>>>> various URNs it knows and cares about is probably a pretty low bar.
>>>> Flink could serve up things it would rather get back untouched in a
>>>> transform with a special flink runner urn.
>>>>
>>>> As Ahmet mentions, SDF is better solution. I hope it's not that far
>>>> away, but even once it comes we'll likely want the above framework to
>>>> invoke the full suite of Java IOs even after they're running on SDF
>>>> themselves.
>>>>
>>>> - Robert
>>>>
>>>> On 31.01.19 17:36, Thomas Weise wrote:
>>>> > Exactly, that's what I had in mind.
>>>> >
>>>> > A Flink runner native transform would make the existing unbounded
>>>> sources
>>>> > available, similar to:
>>>> >
>>>> >
>>>> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>>>> >
>>>> >
>>>> >
>>>> >
>>>> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
>>>> <mxm@apache.org <ma...@apache.org>
>>>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>>>> >
>>>> > Wouldn't it be even more useful for the transition period if
>>>> we enabled Beam IO
>>>> > to be used via Flink (like in the legacy Flink Runner)? In
>>>> this particular
>>>> > example, Matthias wants to use PubSubIO, which is not even
>>>> available as a
>>>> > native
>>>> > Flink transform.
>>>> >
>>>> > On 31.01.19 16:21, Thomas Weise wrote:
>>>> > > Until SDF is supported, we could also add Flink runner
>>>> native transforms for
>>>> > > selected unbounded sources [1].
>>>> > >
>>>> > > That might be a reasonable option to unblock users that
>>>> want to try Python
>>>> > > streaming on Flink.
>>>> > >
>>>> > > Thomas
>>>> > >
>>>> > > [1]
>>>> > >
>>>> >
>>>> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>>>> > >
>>>> > >
>>>> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
>>>> <mxm@apache.org <ma...@apache.org>
>>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>>>> > >
>>>> > > > I have a hard time to imagine how can we map in a
>>>> generic way
>>>> > > RestrictionTrackers into the existing
>>>> Bounded/UnboundedSource, so I would
>>>> > > love to hear more about the details.
>>>> > >
>>>> > > Isn't it the other way around? The SDF is a
>>>> generalization of
>>>> > UnboundedSource.
>>>> > > So we would wrap UnboundedSource using SDF. I'm not
>>>> saying it is
>>>> > trivial, but
>>>> > > SDF offers all the functionality that UnboundedSource
>>>> needs.
>>>> > >
>>>> > > For example, the @GetInitialRestriction method would
>>>> call split on the
>>>> > > UnboundedSource and the restriction trackers would
>>>> then be used to
>>>> > process the
>>>> > > splits.
>>>> > >
>>>> > > On 31.01.19 15:16, Ismaël Mejía wrote:
>>>> > > >> Not necessarily. This would be one way. Another
>>>> way is build an SDF
>>>> > > wrapper for UnboundedSource. Probably the easier path
>>>> for migration.
>>>> > > >
>>>> > > > That would be fantastic, I have heard about such
>>>> wrapper multiple
>>>> > > > times but so far there is not any realistic
>>>> proposal. I have a hard
>>>> > > > time to imagine how can we map in a generic way
>>>> RestrictionTrackers
>>>> > > > into the existing Bounded/UnboundedSource, so I
>>>> would love to hear
>>>> > > > more about the details.
>>>> > > >
>>>> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels
>>>> <mxm@apache.org <ma...@apache.org>
>>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>>>> > > >>
>>>> > > >> > In addition to have support in the runners,
>>>> this will require a
>>>> > > >> > rewrite of PubsubIO to use the new SDF API.
>>>> > > >>
>>>> > > >> Not necessarily. This would be one way. Another
>>>> way is build an SDF
>>>> > > wrapper for
>>>> > > >> UnboundedSource. Probably the easier path for
>>>> migration.
>>>> > > >>
>>>> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
>>>> > > >>>> Fortunately, there is already a pending PR for
>>>> cross-language
>>>> > > pipelines which
>>>> > > >>>> will allow us to use Java IO like PubSub in
>>>> Python jobs.
>>>> > > >>>
>>>> > > >>> In addition to have support in the runners, this
>>>> will require a
>>>> > > >>> rewrite of PubsubIO to use the new SDF API.
>>>> > > >>>
>>>> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
>>>> > <mxm@apache.org <ma...@apache.org>
>>>> <mailto:mxm@apache.org <ma...@apache.org>>
>>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>>> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>>>> > > >>>>
>>>> > > >>>> Hi Matthias,
>>>> > > >>>>
>>>> > > >>>> This is already reflected in the compatibility
>>>> matrix, if you look
>>>> > > under SDF.
>>>> > > >>>> There is no UnboundedSource interface for
>>>> portable pipelines.
>>>> > That's a
>>>> > > legacy
>>>> > > >>>> abstraction that will be replaced with SDF.
>>>> > > >>>>
>>>> > > >>>> Fortunately, there is already a pending PR for
>>>> cross-language
>>>> > > pipelines which
>>>> > > >>>> will allow us to use Java IO like PubSub in
>>>> Python jobs.
>>>> > > >>>>
>>>> > > >>>> Thanks,
>>>> > > >>>> Max
>>>> > > >>>>
>>>> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
>>>> > > >>>>> Hey Ankur,
>>>> > > >>>>>
>>>> > > >>>>> Thanks for the swift reply. Should I change
>>>> this in the
>>>> > capability matrix
>>>> > > >>>>>
>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
>>>> > > >>>>>
>>>> > > >>>>> Many thanks.
>>>> > > >>>>> Best,
>>>> > > >>>>> Matthias
>>>> > > >>>>>
>>>> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
>>>> <goenka@google.com <ma...@google.com>
>>>> > <mailto:goenka@google.com <ma...@google.com>>
>>>> > > <mailto:goenka@google.com <ma...@google.com>
>>>> <mailto:goenka@google.com <ma...@google.com>>>
>>>> > > >>>>> <mailto:goenka@google.com
>>>> <ma...@google.com> <mailto:goenka@google.com
>>>> <ma...@google.com>>
>>>> > <mailto:goenka@google.com <ma...@google.com>
>>>> <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
>>>> > > >>>>>
>>>> > > >>>>> Hi Matthias,
>>>> > > >>>>>
>>>> > > >>>>> Unfortunately, unbounded reads including
>>>> pubsub are not yet
>>>> > > supported for
>>>> > > >>>>> portable runners.
>>>> > > >>>>>
>>>> > > >>>>> Thanks,
>>>> > > >>>>> Ankur
>>>> > > >>>>>
>>>> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias
>>>> Baetens
>>>> > > <baetensmatthias@gmail.com
>>>> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>>
>>>> > <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>>>
>>>> > > >>>>> <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>
>>>> > <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>>
>>>> > > <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>
>>>> > <mailto:baetensmatthias@gmail.com
>>>> <ma...@gmail.com>>>>> wrote:
>>>> > > >>>>>
>>>> > > >>>>> Hi everyone,
>>>> > > >>>>>
>>>> > > >>>>> Last few days I have been trying to
>>>> run a streaming
>>>> > > pipeline (code on
>>>> > > >>>>> Github
>>>> <https://github.com/matthiasa4/beam-demo>) on a
>>>> > > Flink Runner.
>>>> > > >>>>>
>>>> > > >>>>> I am running a Flink cluster locally
>>>> (v1.5.6
>>>> > > >>>>>
>>>> <https://flink.apache.org/downloads.html>)
>>>> > > >>>>> I have built the SDK Harness
>>>> Container: /./gradlew
>>>> > > >>>>> :beam-sdks-python-container:docker/
>>>> > > >>>>> and started the JobServer: /./gradlew
>>>> > > >>>>>
>>>> :beam-runners-flink_2.11-job-server:runShadow
>>>> > > >>>>> -PflinkMasterUrl=localhost:8081./
>>>> > > >>>>>
>>>> > > >>>>> I run my pipeline with: /env/bin/python
>>>> > streaming_pipeline.py
>>>> > > >>>>> --runner=PortableRunner
>>>> --job_endpoint=localhost:8099
>>>> > > --output xxx
>>>> > > >>>>> --input_subscription xxx
>>>> --output_subscription xxx/
>>>> > > >>>>> /
>>>> > > >>>>> /
>>>> > > >>>>> All this is running inside a Ubuntu
>>>> (Bionic) in a
>>>> > Virtualbox.
>>>> > > >>>>>
>>>> > > >>>>> The job submits fine, but
>>>> unfortunately fails after
>>>> > a few
>>>> > > seconds with
>>>> > > >>>>> the error attached.
>>>> > > >>>>>
>>>> > > >>>>> Anything I am missing or doing wrong?
>>>> > > >>>>>
>>>> > > >>>>> Many thanks.
>>>> > > >>>>> Best,
>>>> > > >>>>> Matthias
>>>> > > >>>>>
>>>> > > >>>>>
>>>> > >
>>>> >
>>>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Thanks for the explanation Robert it makes much more sense now. (Sorry
for the confusion in the mapping I mistyped the direction SDF <->
Source).
Status of SDF:
- Support for Dynamic Work Rebalancing is WIP.
- Bounded version translation is supported by all non-portable runners
in a relatively naive way.
- Unbounded version translation is not supported in the non-portable
runners. (Let's not forget that this case may make sense too).
- Portable runners translation of SDF is WIP
- There is only one IO that is written based on SDF:
- HBaseIO
- Some other IOs should work out of the box (those based on
non-splittable DoFn):
- ClickhouseIO
- File-based ones: TextIO, AvroIO, ParquetIO
- JdbcIO
- SolrIO
Max thanks for your summary. I would like to add that we agree that
the runner specific translation via URN is a temporal solution until
the wrappers transforms are written, is this correct? In any case this
alternative standard expansion approach deserves a discussion of their
own as you mention.
On Fri, Feb 1, 2019 at 2:02 PM Robert Bradshaw <ro...@google.com> wrote:
>
> Are you suggesting something akin to a generic
>
> urn: JsonConfiguredJavaSource
> payload: some json specifying which source and which parameters
>
> which would expand to actually constructing and applying that source?
>
> (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi protos that fully specified it, and we use that same format to translate back out.)
>
> On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> Recaping here:
>>
>> We all agree that SDF is the way to go for future implementations of
>> sources. It enables us to get rid of the source interfaces. However, SDF
>> does not solve the lack of streaming sources in Python.
>>
>> The expansion PR (thanks btw!) solves the problem of
>> expanding/translating URNs known to an ExpansionService. That is a more
>> programmatic way of replacing language-specific performs, instead of
>> relying on translators directly in the Runner.
>>
>> What is unsolved is the configuration of sources from a foreign
>> environment. In my opinion this is the most pressing issue for Python
>> sources, because what is PubSubIO worth in Python if you cannot
>> configure it?
>>
>> What about this:
>>
>> I think it is worth adding a JSON configuration option for all existing
>> Java sources. That way, we could easily configure them as part of the
>> expansion request (which would contain a JSON configuration). I'll
>> probably fork a thread to discuss this in more detail, but would like to
>> hear your thoughts.
>>
>> -Max
>>
>> On 01.02.19 13:08, Robert Bradshaw wrote:
>> > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Ah, I thought you meant native Flink transforms.
>> >
>> > Exactly! The translation code is already there. The main challenge
>> > is how to
>> > programmatically configure the BeamIO from Python. I suppose that is
>> > also an
>> > unsolved problem for cross-language transforms in general.
>> >
>> >
>> > This is what https://github.com/apache/beam/pull/7316 does.
>> >
>> > For a particular source, one would want to define a URN and
>> > corresponding payload, then (probably) a CompositeTransform in Python
>> > that takes the users arguments, packages them into the payload, applies
>> > the ExternalTransform, and returns the results. How to handle arbitrary
>> > UDFs embedded in sources is still TBD.
>> >
>> > For Matthias' pipeline with PubSubIO we can build something
>> > specific, but for
>> > the general case there should be way to initialize a Beam IO via a
>> > configuration
>> > map provided by an external environment.
>> >
>> >
>> > I thought quite a bit about how we could represent expansions statically
>> > (e.g. have some kind of expansion template that could be used, at least
>> > in many cases, as data without firing up a separate process. May be
>> > worth doing eventually, but we run into the same issues that were
>> > discussed at
>> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>> >
>> > If one is already using a portable runner like Flink, having the job
>> > service process automatically also serve up an expansion service for
>> > various URNs it knows and cares about is probably a pretty low bar.
>> > Flink could serve up things it would rather get back untouched in a
>> > transform with a special flink runner urn.
>> >
>> > As Ahmet mentions, SDF is better solution. I hope it's not that far
>> > away, but even once it comes we'll likely want the above framework to
>> > invoke the full suite of Java IOs even after they're running on SDF
>> > themselves.
>> >
>> > - Robert
>> >
>> > On 31.01.19 17:36, Thomas Weise wrote:
>> > > Exactly, that's what I had in mind.
>> > >
>> > > A Flink runner native transform would make the existing unbounded
>> > sources
>> > > available, similar to:
>> > >
>> > >
>> > https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>> > >
>> > >
>> > >
>> > >
>> > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
>> > <mxm@apache.org <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>> > >
>> > > Wouldn't it be even more useful for the transition period if
>> > we enabled Beam IO
>> > > to be used via Flink (like in the legacy Flink Runner)? In
>> > this particular
>> > > example, Matthias wants to use PubSubIO, which is not even
>> > available as a
>> > > native
>> > > Flink transform.
>> > >
>> > > On 31.01.19 16:21, Thomas Weise wrote:
>> > > > Until SDF is supported, we could also add Flink runner
>> > native transforms for
>> > > > selected unbounded sources [1].
>> > > >
>> > > > That might be a reasonable option to unblock users that
>> > want to try Python
>> > > > streaming on Flink.
>> > > >
>> > > > Thomas
>> > > >
>> > > > [1]
>> > > >
>> > >
>> > https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>> > > >
>> > > >
>> > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
>> > <mxm@apache.org <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> > > >
>> > > > > I have a hard time to imagine how can we map in a
>> > generic way
>> > > > RestrictionTrackers into the existing
>> > Bounded/UnboundedSource, so I would
>> > > > love to hear more about the details.
>> > > >
>> > > > Isn't it the other way around? The SDF is a
>> > generalization of
>> > > UnboundedSource.
>> > > > So we would wrap UnboundedSource using SDF. I'm not
>> > saying it is
>> > > trivial, but
>> > > > SDF offers all the functionality that UnboundedSource
>> > needs.
>> > > >
>> > > > For example, the @GetInitialRestriction method would
>> > call split on the
>> > > > UnboundedSource and the restriction trackers would
>> > then be used to
>> > > process the
>> > > > splits.
>> > > >
>> > > > On 31.01.19 15:16, Ismaël Mejía wrote:
>> > > > >> Not necessarily. This would be one way. Another
>> > way is build an SDF
>> > > > wrapper for UnboundedSource. Probably the easier path
>> > for migration.
>> > > > >
>> > > > > That would be fantastic, I have heard about such
>> > wrapper multiple
>> > > > > times but so far there is not any realistic
>> > proposal. I have a hard
>> > > > > time to imagine how can we map in a generic way
>> > RestrictionTrackers
>> > > > > into the existing Bounded/UnboundedSource, so I
>> > would love to hear
>> > > > > more about the details.
>> > > > >
>> > > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels
>> > <mxm@apache.org <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> > > > >>
>> > > > >> > In addition to have support in the runners,
>> > this will require a
>> > > > >> > rewrite of PubsubIO to use the new SDF API.
>> > > > >>
>> > > > >> Not necessarily. This would be one way. Another
>> > way is build an SDF
>> > > > wrapper for
>> > > > >> UnboundedSource. Probably the easier path for
>> > migration.
>> > > > >>
>> > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
>> > > > >>>> Fortunately, there is already a pending PR for
>> > cross-language
>> > > > pipelines which
>> > > > >>>> will allow us to use Java IO like PubSub in
>> > Python jobs.
>> > > > >>>
>> > > > >>> In addition to have support in the runners, this
>> > will require a
>> > > > >>> rewrite of PubsubIO to use the new SDF API.
>> > > > >>>
>> > > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
>> > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> > > > >>>>
>> > > > >>>> Hi Matthias,
>> > > > >>>>
>> > > > >>>> This is already reflected in the compatibility
>> > matrix, if you look
>> > > > under SDF.
>> > > > >>>> There is no UnboundedSource interface for
>> > portable pipelines.
>> > > That's a
>> > > > legacy
>> > > > >>>> abstraction that will be replaced with SDF.
>> > > > >>>>
>> > > > >>>> Fortunately, there is already a pending PR for
>> > cross-language
>> > > > pipelines which
>> > > > >>>> will allow us to use Java IO like PubSub in
>> > Python jobs.
>> > > > >>>>
>> > > > >>>> Thanks,
>> > > > >>>> Max
>> > > > >>>>
>> > > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
>> > > > >>>>> Hey Ankur,
>> > > > >>>>>
>> > > > >>>>> Thanks for the swift reply. Should I change
>> > this in the
>> > > capability matrix
>> > > > >>>>>
>> > <https://s.apache.org/apache-beam-portability-support-table> then?
>> > > > >>>>>
>> > > > >>>>> Many thanks.
>> > > > >>>>> Best,
>> > > > >>>>> Matthias
>> > > > >>>>>
>> > > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
>> > <goenka@google.com <ma...@google.com>
>> > > <mailto:goenka@google.com <ma...@google.com>>
>> > > > <mailto:goenka@google.com <ma...@google.com>
>> > <mailto:goenka@google.com <ma...@google.com>>>
>> > > > >>>>> <mailto:goenka@google.com
>> > <ma...@google.com> <mailto:goenka@google.com
>> > <ma...@google.com>>
>> > > <mailto:goenka@google.com <ma...@google.com>
>> > <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
>> > > > >>>>>
>> > > > >>>>> Hi Matthias,
>> > > > >>>>>
>> > > > >>>>> Unfortunately, unbounded reads including
>> > pubsub are not yet
>> > > > supported for
>> > > > >>>>> portable runners.
>> > > > >>>>>
>> > > > >>>>> Thanks,
>> > > > >>>>> Ankur
>> > > > >>>>>
>> > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias
>> > Baetens
>> > > > <baetensmatthias@gmail.com
>> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>>
>> > > <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>>>
>> > > > >>>>> <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>
>> > > <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>>
>> > > > <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>
>> > > <mailto:baetensmatthias@gmail.com
>> > <ma...@gmail.com>>>>> wrote:
>> > > > >>>>>
>> > > > >>>>> Hi everyone,
>> > > > >>>>>
>> > > > >>>>> Last few days I have been trying to
>> > run a streaming
>> > > > pipeline (code on
>> > > > >>>>> Github
>> > <https://github.com/matthiasa4/beam-demo>) on a
>> > > > Flink Runner.
>> > > > >>>>>
>> > > > >>>>> I am running a Flink cluster locally
>> > (v1.5.6
>> > > > >>>>>
>> > <https://flink.apache.org/downloads.html>)
>> > > > >>>>> I have built the SDK Harness
>> > Container: /./gradlew
>> > > > >>>>> :beam-sdks-python-container:docker/
>> > > > >>>>> and started the JobServer: /./gradlew
>> > > > >>>>>
>> > :beam-runners-flink_2.11-job-server:runShadow
>> > > > >>>>> -PflinkMasterUrl=localhost:8081./
>> > > > >>>>>
>> > > > >>>>> I run my pipeline with: /env/bin/python
>> > > streaming_pipeline.py
>> > > > >>>>> --runner=PortableRunner
>> > --job_endpoint=localhost:8099
>> > > > --output xxx
>> > > > >>>>> --input_subscription xxx
>> > --output_subscription xxx/
>> > > > >>>>> /
>> > > > >>>>> /
>> > > > >>>>> All this is running inside a Ubuntu
>> > (Bionic) in a
>> > > Virtualbox.
>> > > > >>>>>
>> > > > >>>>> The job submits fine, but
>> > unfortunately fails after
>> > > a few
>> > > > seconds with
>> > > > >>>>> the error attached.
>> > > > >>>>>
>> > > > >>>>> Anything I am missing or doing wrong?
>> > > > >>>>>
>> > > > >>>>> Many thanks.
>> > > > >>>>> Best,
>> > > > >>>>> Matthias
>> > > > >>>>>
>> > > > >>>>>
>> > > >
>> > >
>> >
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
On Tue, Feb 5, 2019 at 5:11 PM Maximilian Michels <mx...@apache.org> wrote:
>
> Good points Cham.
>
> JSON seemed like the most intuitive way to specify a configuration map.
> We already use JSON in other places, e.g. to specify the environment
> configuration. It is not necessarily a contradiction to have JSON inside
> Protobuf. From the perspective of IO authors, the user-friendliness
> plays a role because they wouldn't have to deal with Protobuf.
>
> I agree that the configuration format is an implementation detail that
> will be hidden to users via easy-to-use wrappers.
JSON has the advantage that one need not know (let alone compile) the
schema to use it. I see this as a particular advantage for a "generic"
layer where the client SDK may not know all the transforms a server
may be serving. It's less useful when the client SDK already knows
about a source and has nice wrappers for it. (Nicer and less
error-prone than manually constructing JSON at least.)
For a particular source, its choice of payload is specific to that
transform. Protos are natural, and could arguably be encouraged for
things we ship with Beam, but by no means required (and, for example,
DoFn's payloads are serialized as raw bytes, not protos).
> Do we have to support UDFs for expanding existing IO? Users would still
> be able to apply UDFs via ParDo on the IO output collections. Generally
> speaking, I can see how for cross-language transforms UDF support would
> be good. For example, a Combine implementation in Java, where the
> combine UDFs come from Python.
The idea of URNs is that one would develop a large body of URNs that
are supported in many languages (e.g. SumInt64s) and then the runner
could pick the environment that goes best (e.g. according to
performance and/or opportunity for fusion or possibly inlining).
UDFs that are called from within an IO as part of its operation is
still an open question.
> I suppose the question is, do we try to solve the general case, or do we
> go with a simpler approach for enabling the use of existing IO first?
> Lack of IO seems to be the most pressing issue for the adoption of Beam
> Python. I imagine that a backwards-compatible incremental support for
> cross-language transforms (IOs first, later other transforms) would be
> possible.
I think what we have is backwards compatible. One can define and
register as many (urn, payload -> PTransform) pairs as one wants.
> On 05.02.19 03:07, Chamikara Jayalath wrote:
> >
> >
> > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Yes, I imagine sources to implement a JsonConfigurable interface (e.g.
> > on their builders):
> >
> > JsonConfigurable {
> > // Either a json string or Map<String, String>
> > apply(String jsonConfig);
> > }
> >
> > In Python we would create this transform:
> >
> > URN: JsonConfiguredSource:v1
> > payload: {
> > environment: environment_id, // Java/Python/Go
> > resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
> > configuration: json config, // { "topic" : "my_pubsub_topic" }
> > }
> >
> >
> > Thanks Max, this is a great first step towards defining to API for
> > cross-language transforms.
> > Is there a reason why you would want to use JSON instead of a proto
> > here. I guess we'll be providing a more user friendly language wrapper
> > (for example, Python) for end-users here, so user-friendliness-wise, the
> > format we choose won't matter much (for pipeline authors).
> > If we don't support UDFs, performance difference will be negligible, but
> > UDFs might require a callback to original SDK (per-element worst case).
> > So might make sense to choose the more efficient format.
> >
> > Also, probably we need to define a more expanded definition (proto/JSON)
> > to support UDFs. For example, a payload + a set of parameter definitions
> > so that the target SDK (for example, Java) can call back the original
> > SDK where the pipeline was authored in (for example, Python) to resolve
> > UDFs at runtime.
> >
> > Thanks,
> > Cham
> >
> > That's more generic and could be used for other languages where we
> > might
> > have sources/sinks.
> >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi protos that fully specified it, and we use that same
> > format to translate back out.)
> >
> > Not that I know of.
> >
> > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > Are you suggesting something akin to a generic
> > >
> > > urn: JsonConfiguredJavaSource
> > > payload: some json specifying which source and which parameters
> > >
> > > which would expand to actually constructing and applying that source?
> > >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi
> > > protos that fully specified it, and we use that same format to
> > translate
> > > back out.)
> > >
> > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Recaping here:
> > >
> > > We all agree that SDF is the way to go for future
> > implementations of
> > > sources. It enables us to get rid of the source interfaces.
> > However,
> > > SDF
> > > does not solve the lack of streaming sources in Python.
> > >
> > > The expansion PR (thanks btw!) solves the problem of
> > > expanding/translating URNs known to an ExpansionService. That
> > is a more
> > > programmatic way of replacing language-specific performs,
> > instead of
> > > relying on translators directly in the Runner.
> > >
> > > What is unsolved is the configuration of sources from a foreign
> > > environment. In my opinion this is the most pressing issue
> > for Python
> > > sources, because what is PubSubIO worth in Python if you cannot
> > > configure it?
> > >
> > > What about this:
> > >
> > > I think it is worth adding a JSON configuration option for
> > all existing
> > > Java sources. That way, we could easily configure them as
> > part of the
> > > expansion request (which would contain a JSON configuration).
> > I'll
> > > probably fork a thread to discuss this in more detail, but would
> > > like to
> > > hear your thoughts.
> > >
> > > -Max
> > >
> > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Ah, I thought you meant native Flink transforms.
> > > >
> > > > Exactly! The translation code is already there. The main
> > > challenge
> > > > is how to
> > > > programmatically configure the BeamIO from Python. I
> > suppose
> > > that is
> > > > also an
> > > > unsolved problem for cross-language transforms in general.
> > > >
> > > >
> > > > This is what https://github.com/apache/beam/pull/7316 does.
> > > >
> > > > For a particular source, one would want to define a URN and
> > > > corresponding payload, then (probably) a CompositeTransform in
> > > Python
> > > > that takes the users arguments, packages them into the
> > payload,
> > > applies
> > > > the ExternalTransform, and returns the results. How to handle
> > > arbitrary
> > > > UDFs embedded in sources is still TBD.
> > > >
> > > > For Matthias' pipeline with PubSubIO we can build
> > something
> > > > specific, but for
> > > > the general case there should be way to initialize a
> > Beam IO
> > > via a
> > > > configuration
> > > > map provided by an external environment.
> > > >
> > > >
> > > > I thought quite a bit about how we could represent expansions
> > > statically
> > > > (e.g. have some kind of expansion template that could be
> > used, at
> > > least
> > > > in many cases, as data without firing up a separate
> > process. May be
> > > > worth doing eventually, but we run into the same issues
> > that were
> > > > discussed at
> > > >
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > >
> > > > If one is already using a portable runner like Flink,
> > having the job
> > > > service process automatically also serve up an expansion
> > service for
> > > > various URNs it knows and cares about is probably a pretty
> > low bar.
> > > > Flink could serve up things it would rather get back
> > untouched in a
> > > > transform with a special flink runner urn.
> > > >
> > > > As Ahmet mentions, SDF is better solution. I hope it's not
> > that far
> > > > away, but even once it comes we'll likely want the above
> > > framework to
> > > > invoke the full suite of Java IOs even after they're
> > running on SDF
> > > > themselves.
> > > >
> > > > - Robert
> > > >
> > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > Exactly, that's what I had in mind.
> > > > >
> > > > > A Flink runner native transform would make the existing
> > > unbounded
> > > > sources
> > > > > available, similar to:
> > > > >
> > > > >
> > > >
> > >
> > https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Wouldn't it be even more useful for the transition
> > > period if
> > > > we enabled Beam IO
> > > > > to be used via Flink (like in the legacy Flink
> > Runner)? In
> > > > this particular
> > > > > example, Matthias wants to use PubSubIO, which
> > is not even
> > > > available as a
> > > > > native
> > > > > Flink transform.
> > > > >
> > > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > > Until SDF is supported, we could also add
> > Flink runner
> > > > native transforms for
> > > > > > selected unbounded sources [1].
> > > > > >
> > > > > > That might be a reasonable option to unblock
> > users that
> > > > want to try Python
> > > > > > streaming on Flink.
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> > https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian
> > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > > I have a hard time to imagine how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers into the existing
> > > > Bounded/UnboundedSource, so I would
> > > > > > love to hear more about the details.
> > > > > >
> > > > > > Isn't it the other way around? The SDF is a
> > > > generalization of
> > > > > UnboundedSource.
> > > > > > So we would wrap UnboundedSource using
> > SDF. I'm not
> > > > saying it is
> > > > > trivial, but
> > > > > > SDF offers all the functionality that
> > > UnboundedSource
> > > > needs.
> > > > > >
> > > > > > For example, the @GetInitialRestriction
> > method
> > > would
> > > > call split on the
> > > > > > UnboundedSource and the restriction
> > trackers would
> > > > then be used to
> > > > > process the
> > > > > > splits.
> > > > > >
> > > > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > > > >> Not necessarily. This would be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for UnboundedSource. Probably the
> > > easier path
> > > > for migration.
> > > > > > >
> > > > > > > That would be fantastic, I have heard
> > about such
> > > > wrapper multiple
> > > > > > > times but so far there is not any
> > realistic
> > > > proposal. I have a hard
> > > > > > > time to imagine how can we map in a
> > generic way
> > > > RestrictionTrackers
> > > > > > > into the existing
> > Bounded/UnboundedSource, so I
> > > > would love to hear
> > > > > > > more about the details.
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 3:07 PM
> > Maximilian
> > > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > > >>
> > > > > > >> > In addition to have support in
> > the runners,
> > > > this will require a
> > > > > > >> > rewrite of PubsubIO to use the
> > new SDF API.
> > > > > > >>
> > > > > > >> Not necessarily. This would be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for
> > > > > > >> UnboundedSource. Probably the easier
> > path for
> > > > migration.
> > > > > > >>
> > > > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>
> > > > > > >>> In addition to have support in the
> > > runners, this
> > > > will require a
> > > > > > >>> rewrite of PubsubIO to use the new
> > SDF API.
> > > > > > >>>
> > > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > > Maximilian Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > > >>>>
> > > > > > >>>> Hi Matthias,
> > > > > > >>>>
> > > > > > >>>> This is already reflected in the
> > > compatibility
> > > > matrix, if you look
> > > > > > under SDF.
> > > > > > >>>> There is no UnboundedSource
> > interface for
> > > > portable pipelines.
> > > > > That's a
> > > > > > legacy
> > > > > > >>>> abstraction that will be replaced
> > with SDF.
> > > > > > >>>>
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> Max
> > > > > > >>>>
> > > > > > >>>> On 31.01.19 12:06, Matthias
> > Baetens wrote:
> > > > > > >>>>> Hey Ankur,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the swift reply.
> > Should I change
> > > > this in the
> > > > > capability matrix
> > > > > > >>>>>
> > > >
> > <https://s.apache.org/apache-beam-portability-support-table>
> > > then?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>> On Thu, 31 Jan 2019 at 09:31,
> > Ankur Goenka
> > > > <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > > <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi Matthias,
> > > > > > >>>>>
> > > > > > >>>>> Unfortunately, unbounded reads
> > > including
> > > > pubsub are not yet
> > > > > > supported for
> > > > > > >>>>> portable runners.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>> Ankur
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> > > Matthias
> > > > Baetens
> > > > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>
> > > > > > >>>>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi everyone,
> > > > > > >>>>>
> > > > > > >>>>> Last few days I have been
> > > trying to
> > > > run a streaming
> > > > > > pipeline (code on
> > > > > > >>>>> Github
> > > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > > Flink Runner.
> > > > > > >>>>>
> > > > > > >>>>> I am running a Flink
> > cluster
> > > locally
> > > > (v1.5.6
> > > > > > >>>>>
> > > > <https://flink.apache.org/downloads.html>)
> > > > > > >>>>> I have built the SDK
> > Harness
> > > > Container: /./gradlew
> > > > > > >>>>>
> > > :beam-sdks-python-container:docker/
> > > > > > >>>>> and started the JobServer:
> > > /./gradlew
> > > > > > >>>>>
> > > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > > >>>>>
> > -PflinkMasterUrl=localhost:8081./
> > > > > > >>>>>
> > > > > > >>>>> I run my pipeline with:
> > > /env/bin/python
> > > > > streaming_pipeline.py
> > > > > > >>>>> --runner=PortableRunner
> > > > --job_endpoint=localhost:8099
> > > > > > --output xxx
> > > > > > >>>>> --input_subscription xxx
> > > > --output_subscription xxx/
> > > > > > >>>>> /
> > > > > > >>>>> /
> > > > > > >>>>> All this is running
> > inside a
> > > Ubuntu
> > > > (Bionic) in a
> > > > > Virtualbox.
> > > > > > >>>>>
> > > > > > >>>>> The job submits fine, but
> > > > unfortunately fails after
> > > > > a few
> > > > > > seconds with
> > > > > > >>>>> the error attached.
> > > > > > >>>>>
> > > > > > >>>>> Anything I am missing
> > or doing
> > > wrong?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
Thanks for writing this up. I'd like to point out that though this seems to
be specifically targeting IOs, there's nothing here that is specific to
IOs.
On Thu, Feb 7, 2019 at 9:30 PM Chamikara Jayalath <ch...@google.com>
wrote:
> Thanks Max. Added few comments.
>
> - Cham
>
> On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
>
>> Can you please add the link to the design document webpage.
>>
>> Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels <mx...@apache.org> a
>> écrit :
>>
>>> I've created an initial design document:
>>> https://s.apache.org/beam-cross-language-io
>>>
>>> It does not contain all the details but perhaps it's a good basis for a
>>> discussion on how we proceed.
>>>
>>> -Max
>>>
>>> On 06.02.19 19:49, Chamikara Jayalath wrote:
>>> >
>>> >
>>> > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <mxm@apache.org
>>> > <ma...@apache.org>> wrote:
>>> >
>>> > Thanks for your replies Robert and Cham.
>>> >
>>> > What I had in mind was a generic Wrapper that would easily allow
>>> users
>>> > to use IO from Java. Such wrapper could start as an experimental
>>> > feature
>>> > and then, through URN versioning, become stable eventually.
>>> >
>>> > UDFs are needed, though they are a special case. Most users
>>> (including
>>> > Matthias) just want to specify a few String options which do not
>>> > require
>>> > UDFs but something along the lines what I proposed here.
>>> >
>>> >
>>> > Sounds good let's start documenting/implementing the "easy" case and
>>> > think bit more regarding UDFs.
>>> >
>>> >
>>> >
>>> > Robert wrote:
>>> > > UDFs that are called from within an IO as part of its operation
>>> is
>>> > > still an open question.
>>> >
>>> > Exactly. How about we solve the easier case first, unblock users,
>>> and
>>> > then think more about solving the general case?
>>> >
>>> > Cham wrote:
>>> > > I'm happy to work with you to realize this.
>>> >
>>> > Would be great to exchange more ideas on this! I can compile the
>>> > current
>>> > ideas we have in a document and we move from there.
>>> >
>>> >
>>> > Great. Thanks.
>>> >
>>> >
>>> > Thanks,
>>> > Max
>>> >
>>> > On 05.02.19 17:56, Chamikara Jayalath wrote:
>>> > >
>>> > >
>>> > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>> wrote:
>>> > >
>>> > > Good points Cham.
>>> > >
>>> > > JSON seemed like the most intuitive way to specify a
>>> > configuration map.
>>> > > We already use JSON in other places, e.g. to specify the
>>> > environment
>>> > > configuration. It is not necessarily a contradiction to
>>> have JSON
>>> > > inside
>>> > > Protobuf. From the perspective of IO authors, the
>>> > user-friendliness
>>> > > plays a role because they wouldn't have to deal with
>>> Protobuf.
>>> > >
>>> > >
>>> > > It's a good point that JSON will make this more user-friendly
>>> for IO
>>> > > transforms authors. Probably we should do a bit of
>>> > experimentation and
>>> > > keep this experimental in case we hit a performance snag.
>>> > >
>>> > >
>>> > > I agree that the configuration format is an implementation
>>> > detail that
>>> > > will be hidden to users via easy-to-use wrappers.
>>> > >
>>> > > Do we have to support UDFs for expanding existing IO? Users
>>> > would still
>>> > > be able to apply UDFs via ParDo on the IO output
>>> collections.
>>> > Generally
>>> > > speaking, I can see how for cross-language transforms UDF
>>> > support would
>>> > > be good. For example, a Combine implementation in Java,
>>> where the
>>> > > combine UDFs come from Python.
>>> > >
>>> > >
>>> > > I think we should try to support UDFs in the first version
>>> unless
>>> > > there's a major blocker that hinders realizing this. Many IO
>>> > transforms
>>> > > available today expect users to pass UDFs to realize various
>>> > features
>>> > > (for example, dynamic destinations for BigqueryIO and FileIO,
>>> > timestamp
>>> > > function for KafkaIO). I think without support for UDFs
>>> usability of
>>> > > cross-language transforms feature will be significantly limited.
>>> > >
>>> > > I'm happy to work with you to realize this.
>>> > >
>>> > > Thanks,
>>> > > Cham
>>> > >
>>> > >
>>> > > I suppose the question is, do we try to solve the general
>>> > case, or
>>> > > do we
>>> > > go with a simpler approach for enabling the use of existing
>>> > IO first?
>>> > > Lack of IO seems to be the most pressing issue for the
>>> > adoption of Beam
>>> > > Python. I imagine that a backwards-compatible incremental
>>> > support for
>>> > > cross-language transforms (IOs first, later other
>>> transforms)
>>> > would be
>>> > > possible.
>>> > >
>>> > > -Max
>>> > >
>>> > > On 05.02.19 03:07, Chamikara Jayalath wrote:
>>> > > >
>>> > > >
>>> > > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels
>>> > <mxm@apache.org <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>>> > > >
>>> > > > Yes, I imagine sources to implement a
>>> JsonConfigurable
>>> > > interface (e.g.
>>> > > > on their builders):
>>> > > >
>>> > > > JsonConfigurable {
>>> > > > // Either a json string or Map<String, String>
>>> > > > apply(String jsonConfig);
>>> > > > }
>>> > > >
>>> > > > In Python we would create this transform:
>>> > > >
>>> > > > URN: JsonConfiguredSource:v1
>>> > > > payload: {
>>> > > > environment: environment_id, // Java/Python/Go
>>> > > > resourceIdentifier: string, //
>>> > > "org.apache.beam.io.PubSubIO"
>>> > > > configuration: json config, // { "topic" :
>>> > > "my_pubsub_topic" }
>>> > > > }
>>> > > >
>>> > > >
>>> > > > Thanks Max, this is a great first step towards defining
>>> to
>>> > API for
>>> > > > cross-language transforms.
>>> > > > Is there a reason why you would want to use JSON instead
>>> > of a proto
>>> > > > here. I guess we'll be providing a more user friendly
>>> language
>>> > > wrapper
>>> > > > (for example, Python) for end-users here, so
>>> > > user-friendliness-wise, the
>>> > > > format we choose won't matter much (for pipeline
>>> authors).
>>> > > > If we don't support UDFs, performance difference will be
>>> > > negligible, but
>>> > > > UDFs might require a callback to original SDK
>>> (per-element
>>> > worst
>>> > > case).
>>> > > > So might make sense to choose the more efficient format.
>>> > > >
>>> > > > Also, probably we need to define a more expanded
>>> definition
>>> > > (proto/JSON)
>>> > > > to support UDFs. For example, a payload + a set of
>>> parameter
>>> > > definitions
>>> > > > so that the target SDK (for example, Java) can call back
>>> the
>>> > > original
>>> > > > SDK where the pipeline was authored in (for example,
>>> > Python) to
>>> > > resolve
>>> > > > UDFs at runtime.
>>> > > >
>>> > > > Thanks,
>>> > > > Cham
>>> > > >
>>> > > > That's more generic and could be used for other
>>> languages
>>> > > where we
>>> > > > might
>>> > > > have sources/sinks.
>>> > > >
>>> > > > > (FWIW, I was imagining PubSubIO already had a
>>> > translation into
>>> > > > BeamFnApi protos that fully specified it, and we use
>>> > that same
>>> > > > format to translate back out.)
>>> > > >
>>> > > > Not that I know of.
>>> > > >
>>> > > > On 01.02.19 14:02, Robert Bradshaw wrote:
>>> > > > > Are you suggesting something akin to a generic
>>> > > > >
>>> > > > > urn: JsonConfiguredJavaSource
>>> > > > > payload: some json specifying which source
>>> and
>>> > which
>>> > > parameters
>>> > > > >
>>> > > > > which would expand to actually constructing and
>>> > applying
>>> > > that source?
>>> > > > >
>>> > > > > (FWIW, I was imagining PubSubIO already had a
>>> > translation into
>>> > > > BeamFnApi
>>> > > > > protos that fully specified it, and we use that
>>> > same format to
>>> > > > translate
>>> > > > > back out.)
>>> > > > >
>>> > > > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
>>> > > <mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
>>> > > > >
>>> > > > > Recaping here:
>>> > > > >
>>> > > > > We all agree that SDF is the way to go for
>>> future
>>> > > > implementations of
>>> > > > > sources. It enables us to get rid of the
>>> source
>>> > > interfaces.
>>> > > > However,
>>> > > > > SDF
>>> > > > > does not solve the lack of streaming sources
>>> in
>>> > Python.
>>> > > > >
>>> > > > > The expansion PR (thanks btw!) solves the
>>> > problem of
>>> > > > > expanding/translating URNs known to an
>>> > > ExpansionService. That
>>> > > > is a more
>>> > > > > programmatic way of replacing
>>> language-specific
>>> > performs,
>>> > > > instead of
>>> > > > > relying on translators directly in the Runner.
>>> > > > >
>>> > > > > What is unsolved is the configuration of
>>> > sources from
>>> > > a foreign
>>> > > > > environment. In my opinion this is the most
>>> > pressing issue
>>> > > > for Python
>>> > > > > sources, because what is PubSubIO worth in
>>> > Python if
>>> > > you cannot
>>> > > > > configure it?
>>> > > > >
>>> > > > > What about this:
>>> > > > >
>>> > > > > I think it is worth adding a JSON
>>> configuration
>>> > option for
>>> > > > all existing
>>> > > > > Java sources. That way, we could easily
>>> > configure them as
>>> > > > part of the
>>> > > > > expansion request (which would contain a JSON
>>> > > configuration).
>>> > > > I'll
>>> > > > > probably fork a thread to discuss this in more
>>> > detail,
>>> > > but would
>>> > > > > like to
>>> > > > > hear your thoughts.
>>> > > > >
>>> > > > > -Max
>>> > > > >
>>> > > > > On 01.02.19 13:08, Robert Bradshaw wrote:
>>> > > > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian
>>> > Michels
>>> > > > > <mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
>>> > > > > >
>>> > > > > > Ah, I thought you meant native Flink
>>> > transforms.
>>> > > > > >
>>> > > > > > Exactly! The translation code is
>>> already
>>> > there.
>>> > > The main
>>> > > > > challenge
>>> > > > > > is how to
>>> > > > > > programmatically configure the BeamIO
>>> from
>>> > > Python. I
>>> > > > suppose
>>> > > > > that is
>>> > > > > > also an
>>> > > > > > unsolved problem for cross-language
>>> > transforms
>>> > > in general.
>>> > > > > >
>>> > > > > >
>>> > > > > > This is what
>>> > > https://github.com/apache/beam/pull/7316 does.
>>> > > > > >
>>> > > > > > For a particular source, one would want to
>>> > define a
>>> > > URN and
>>> > > > > > corresponding payload, then (probably) a
>>> > > CompositeTransform in
>>> > > > > Python
>>> > > > > > that takes the users arguments, packages
>>> > them into the
>>> > > > payload,
>>> > > > > applies
>>> > > > > > the ExternalTransform, and returns the
>>> > results. How
>>> > > to handle
>>> > > > > arbitrary
>>> > > > > > UDFs embedded in sources is still TBD.
>>> > > > > >
>>> > > > > > For Matthias' pipeline with PubSubIO we
>>> > can build
>>> > > > something
>>> > > > > > specific, but for
>>> > > > > > the general case there should be way to
>>> > > initialize a
>>> > > > Beam IO
>>> > > > > via a
>>> > > > > > configuration
>>> > > > > > map provided by an external
>>> environment.
>>> > > > > >
>>> > > > > >
>>> > > > > > I thought quite a bit about how we could
>>> > represent
>>> > > expansions
>>> > > > > statically
>>> > > > > > (e.g. have some kind of expansion template
>>> that
>>> > > could be
>>> > > > used, at
>>> > > > > least
>>> > > > > > in many cases, as data without firing up a
>>> > separate
>>> > > > process. May be
>>> > > > > > worth doing eventually, but we run into the
>>> > same issues
>>> > > > that were
>>> > > > > > discussed at
>>> > > > > >
>>> > > >
>>> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>>> > > > > >
>>> > > > > > If one is already using a portable runner
>>> > like Flink,
>>> > > > having the job
>>> > > > > > service process automatically also serve
>>> up an
>>> > > expansion
>>> > > > service for
>>> > > > > > various URNs it knows and cares about is
>>> > probably a
>>> > > pretty
>>> > > > low bar.
>>> > > > > > Flink could serve up things it would rather
>>> > get back
>>> > > > untouched in a
>>> > > > > > transform with a special flink runner urn.
>>> > > > > >
>>> > > > > > As Ahmet mentions, SDF is better solution.
>>> I
>>> > hope
>>> > > it's not
>>> > > > that far
>>> > > > > > away, but even once it comes we'll likely
>>> > want the
>>> > > above
>>> > > > > framework to
>>> > > > > > invoke the full suite of Java IOs even
>>> after
>>> > they're
>>> > > > running on SDF
>>> > > > > > themselves.
>>> > > > > >
>>> > > > > > - Robert
>>> > > > > >
>>> > > > > > On 31.01.19 17:36, Thomas Weise wrote:
>>> > > > > > > Exactly, that's what I had in mind.
>>> > > > > > >
>>> > > > > > > A Flink runner native transform
>>> would
>>> > make
>>> > > the existing
>>> > > > > unbounded
>>> > > > > > sources
>>> > > > > > > available, similar to:
>>> > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > >
>>> > > > > > > On Thu, Jan 31, 2019 at 8:18 AM
>>> > Maximilian
>>> > > Michels
>>> > > > > > <mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
>>> > > > > > >
>>> > > > > > > Wouldn't it be even more useful
>>> > for the
>>> > > transition
>>> > > > > period if
>>> > > > > > we enabled Beam IO
>>> > > > > > > to be used via Flink (like in
>>> the
>>> > legacy
>>> > > Flink
>>> > > > Runner)? In
>>> > > > > > this particular
>>> > > > > > > example, Matthias wants to use
>>> > PubSubIO,
>>> > > which
>>> > > > is not even
>>> > > > > > available as a
>>> > > > > > > native
>>> > > > > > > Flink transform.
>>> > > > > > >
>>> > > > > > > On 31.01.19 16:21, Thomas Weise
>>> > wrote:
>>> > > > > > > > Until SDF is supported, we
>>> > could also add
>>> > > > Flink runner
>>> > > > > > native transforms for
>>> > > > > > > > selected unbounded sources
>>> [1].
>>> > > > > > > >
>>> > > > > > > > That might be a reasonable
>>> > option to
>>> > > unblock
>>> > > > users that
>>> > > > > > want to try Python
>>> > > > > > > > streaming on Flink.
>>> > > > > > > >
>>> > > > > > > > Thomas
>>> > > > > > > >
>>> > > > > > > > [1]
>>> > > > > > > >
>>> > > > > > >
>>> > > > > >
>>> > > > >
>>> > > >
>>> > >
>>> >
>>> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>>> > > > > > > >
>>> > > > > > > >
>>> > > > > > > > On Thu, Jan 31, 2019 at 6:51
>>> AM
>>> > > Maximilian
>>> > > > Michels
>>> > > > > > <mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>>> > > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
>>> > > > > > > >
>>> > > > > > > > > I have a hard time to
>>> > imagine
>>> > > how can
>>> > > > we map
>>> > > > > in a
>>> > > > > > generic way
>>> > > > > > > > RestrictionTrackers into
>>> > the existing
>>> > > > > > Bounded/UnboundedSource, so I would
>>> > > > > > > > love to hear more about
>>> > the details.
>>> > > > > > > >
>>> > > > > > > > Isn't it the other way
>>> around?
>>> > > The SDF is a
>>> > > > > > generalization of
>>> > > > > > > UnboundedSource.
>>> > > > > > > > So we would wrap
>>> > UnboundedSource
>>> > > using
>>> > > > SDF. I'm not
>>> > > > > > saying it is
>>> > > > > > > trivial, but
>>> > > > > > > > SDF offers all the
>>> > functionality that
>>> > > > > UnboundedSource
>>> > > > > > needs.
>>> > > > > > > >
>>> > > > > > > > For example, the
>>> > > @GetInitialRestriction
>>> > > > method
>>> > > > > would
>>> > > > > > call split on the
>>> > > > > > > > UnboundedSource and the
>>> > restriction
>>> > > > trackers would
>>> > > > > > then be used to
>>> > > > > > > process the
>>> > > > > > > > splits.
>>> > > > > > > >
>>> > > > > > > > On 31.01.19 15:16, Ismaël
>>> > Mejía
>>> > > wrote:
>>> > > > > > > > >> Not necessarily. This
>>> > would
>>> > > be one
>>> > > > way. Another
>>> > > > > > way is build an SDF
>>> > > > > > > > wrapper for
>>> UnboundedSource.
>>> > > Probably the
>>> > > > > easier path
>>> > > > > > for migration.
>>> > > > > > > > >
>>> > > > > > > > > That would be
>>> fantastic, I
>>> > > have heard
>>> > > > about such
>>> > > > > > wrapper multiple
>>> > > > > > > > > times but so far there
>>> > is not any
>>> > > > realistic
>>> > > > > > proposal. I have a hard
>>> > > > > > > > > time to imagine how
>>> can
>>> > we map
>>> > > in a
>>> > > > generic way
>>> > > > > > RestrictionTrackers
>>> > > > > > > > > into the existing
>>> > > > Bounded/UnboundedSource, so I
>>> > > > > > would love to hear
>>> > > > > > > > > more about the
>>> details.
>>> > > > > > > > >
>>> > > > > > > > > On Thu, Jan 31, 2019
>>> at
>>> > 3:07 PM
>>> > > > Maximilian
>>> > > > > Michels
>>> > > > > > <mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>>> > > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
>>> > > > > > > > >>
>>> > > > > > > > >> > In addition to
>>> have
>>> > > support in
>>> > > > the runners,
>>> > > > > > this will require a
>>> > > > > > > > >> > rewrite of
>>> PubsubIO to
>>> > > use the
>>> > > > new SDF API.
>>> > > > > > > > >>
>>> > > > > > > > >> Not necessarily. This
>>> > would
>>> > > be one
>>> > > > way. Another
>>> > > > > > way is build an SDF
>>> > > > > > > > wrapper for
>>> > > > > > > > >> UnboundedSource.
>>> > Probably the
>>> > > easier
>>> > > > path for
>>> > > > > > migration.
>>> > > > > > > > >>
>>> > > > > > > > >> On 31.01.19 14:03,
>>> Ismaël
>>> > > Mejía wrote:
>>> > > > > > > > >>>> Fortunately, there
>>> > is already a
>>> > > > pending
>>> > > > > PR for
>>> > > > > > cross-language
>>> > > > > > > > pipelines which
>>> > > > > > > > >>>> will allow us to
>>> use
>>> > Java
>>> > > IO like
>>> > > > PubSub in
>>> > > > > > Python jobs.
>>> > > > > > > > >>>
>>> > > > > > > > >>> In addition to have
>>> > support
>>> > > in the
>>> > > > > runners, this
>>> > > > > > will require a
>>> > > > > > > > >>> rewrite of PubsubIO
>>> > to use
>>> > > the new
>>> > > > SDF API.
>>> > > > > > > > >>>
>>> > > > > > > > >>> On Thu, Jan 31, 2019
>>> > at 12:23 PM
>>> > > > > Maximilian Michels
>>> > > > > > > <mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>>> > > > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>>> > > > > > <mailto:mxm@apache.org
>>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>>> mxm@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>>> > > > > <mailto:mxm@apache.org <mailto:mxm@apache.org
>>> >
>>> > <mailto:mxm@apache.org <ma...@apache.org>>
>>> > > <mailto:mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>>> > > > <mail
>>
>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Chamikara Jayalath <ch...@google.com>.
Thanks Max. Added few comments.
- Cham
On Thu, Feb 7, 2019 at 11:18 AM Ismaël Mejía <ie...@gmail.com> wrote:
> Can you please add the link to the design document webpage.
>
> Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels <mx...@apache.org> a
> écrit :
>
>> I've created an initial design document:
>> https://s.apache.org/beam-cross-language-io
>>
>> It does not contain all the details but perhaps it's a good basis for a
>> discussion on how we proceed.
>>
>> -Max
>>
>> On 06.02.19 19:49, Chamikara Jayalath wrote:
>> >
>> >
>> > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Thanks for your replies Robert and Cham.
>> >
>> > What I had in mind was a generic Wrapper that would easily allow
>> users
>> > to use IO from Java. Such wrapper could start as an experimental
>> > feature
>> > and then, through URN versioning, become stable eventually.
>> >
>> > UDFs are needed, though they are a special case. Most users
>> (including
>> > Matthias) just want to specify a few String options which do not
>> > require
>> > UDFs but something along the lines what I proposed here.
>> >
>> >
>> > Sounds good let's start documenting/implementing the "easy" case and
>> > think bit more regarding UDFs.
>> >
>> >
>> >
>> > Robert wrote:
>> > > UDFs that are called from within an IO as part of its operation
>> is
>> > > still an open question.
>> >
>> > Exactly. How about we solve the easier case first, unblock users,
>> and
>> > then think more about solving the general case?
>> >
>> > Cham wrote:
>> > > I'm happy to work with you to realize this.
>> >
>> > Would be great to exchange more ideas on this! I can compile the
>> > current
>> > ideas we have in a document and we move from there.
>> >
>> >
>> > Great. Thanks.
>> >
>> >
>> > Thanks,
>> > Max
>> >
>> > On 05.02.19 17:56, Chamikara Jayalath wrote:
>> > >
>> > >
>> > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>> wrote:
>> > >
>> > > Good points Cham.
>> > >
>> > > JSON seemed like the most intuitive way to specify a
>> > configuration map.
>> > > We already use JSON in other places, e.g. to specify the
>> > environment
>> > > configuration. It is not necessarily a contradiction to have
>> JSON
>> > > inside
>> > > Protobuf. From the perspective of IO authors, the
>> > user-friendliness
>> > > plays a role because they wouldn't have to deal with
>> Protobuf.
>> > >
>> > >
>> > > It's a good point that JSON will make this more user-friendly
>> for IO
>> > > transforms authors. Probably we should do a bit of
>> > experimentation and
>> > > keep this experimental in case we hit a performance snag.
>> > >
>> > >
>> > > I agree that the configuration format is an implementation
>> > detail that
>> > > will be hidden to users via easy-to-use wrappers.
>> > >
>> > > Do we have to support UDFs for expanding existing IO? Users
>> > would still
>> > > be able to apply UDFs via ParDo on the IO output collections.
>> > Generally
>> > > speaking, I can see how for cross-language transforms UDF
>> > support would
>> > > be good. For example, a Combine implementation in Java,
>> where the
>> > > combine UDFs come from Python.
>> > >
>> > >
>> > > I think we should try to support UDFs in the first version unless
>> > > there's a major blocker that hinders realizing this. Many IO
>> > transforms
>> > > available today expect users to pass UDFs to realize various
>> > features
>> > > (for example, dynamic destinations for BigqueryIO and FileIO,
>> > timestamp
>> > > function for KafkaIO). I think without support for UDFs
>> usability of
>> > > cross-language transforms feature will be significantly limited.
>> > >
>> > > I'm happy to work with you to realize this.
>> > >
>> > > Thanks,
>> > > Cham
>> > >
>> > >
>> > > I suppose the question is, do we try to solve the general
>> > case, or
>> > > do we
>> > > go with a simpler approach for enabling the use of existing
>> > IO first?
>> > > Lack of IO seems to be the most pressing issue for the
>> > adoption of Beam
>> > > Python. I imagine that a backwards-compatible incremental
>> > support for
>> > > cross-language transforms (IOs first, later other transforms)
>> > would be
>> > > possible.
>> > >
>> > > -Max
>> > >
>> > > On 05.02.19 03:07, Chamikara Jayalath wrote:
>> > > >
>> > > >
>> > > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels
>> > <mxm@apache.org <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
>> > > >
>> > > > Yes, I imagine sources to implement a JsonConfigurable
>> > > interface (e.g.
>> > > > on their builders):
>> > > >
>> > > > JsonConfigurable {
>> > > > // Either a json string or Map<String, String>
>> > > > apply(String jsonConfig);
>> > > > }
>> > > >
>> > > > In Python we would create this transform:
>> > > >
>> > > > URN: JsonConfiguredSource:v1
>> > > > payload: {
>> > > > environment: environment_id, // Java/Python/Go
>> > > > resourceIdentifier: string, //
>> > > "org.apache.beam.io.PubSubIO"
>> > > > configuration: json config, // { "topic" :
>> > > "my_pubsub_topic" }
>> > > > }
>> > > >
>> > > >
>> > > > Thanks Max, this is a great first step towards defining to
>> > API for
>> > > > cross-language transforms.
>> > > > Is there a reason why you would want to use JSON instead
>> > of a proto
>> > > > here. I guess we'll be providing a more user friendly
>> language
>> > > wrapper
>> > > > (for example, Python) for end-users here, so
>> > > user-friendliness-wise, the
>> > > > format we choose won't matter much (for pipeline authors).
>> > > > If we don't support UDFs, performance difference will be
>> > > negligible, but
>> > > > UDFs might require a callback to original SDK (per-element
>> > worst
>> > > case).
>> > > > So might make sense to choose the more efficient format.
>> > > >
>> > > > Also, probably we need to define a more expanded
>> definition
>> > > (proto/JSON)
>> > > > to support UDFs. For example, a payload + a set of
>> parameter
>> > > definitions
>> > > > so that the target SDK (for example, Java) can call back
>> the
>> > > original
>> > > > SDK where the pipeline was authored in (for example,
>> > Python) to
>> > > resolve
>> > > > UDFs at runtime.
>> > > >
>> > > > Thanks,
>> > > > Cham
>> > > >
>> > > > That's more generic and could be used for other
>> languages
>> > > where we
>> > > > might
>> > > > have sources/sinks.
>> > > >
>> > > > > (FWIW, I was imagining PubSubIO already had a
>> > translation into
>> > > > BeamFnApi protos that fully specified it, and we use
>> > that same
>> > > > format to translate back out.)
>> > > >
>> > > > Not that I know of.
>> > > >
>> > > > On 01.02.19 14:02, Robert Bradshaw wrote:
>> > > > > Are you suggesting something akin to a generic
>> > > > >
>> > > > > urn: JsonConfiguredJavaSource
>> > > > > payload: some json specifying which source and
>> > which
>> > > parameters
>> > > > >
>> > > > > which would expand to actually constructing and
>> > applying
>> > > that source?
>> > > > >
>> > > > > (FWIW, I was imagining PubSubIO already had a
>> > translation into
>> > > > BeamFnApi
>> > > > > protos that fully specified it, and we use that
>> > same format to
>> > > > translate
>> > > > > back out.)
>> > > > >
>> > > > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
>> > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
>> > > > >
>> > > > > Recaping here:
>> > > > >
>> > > > > We all agree that SDF is the way to go for
>> future
>> > > > implementations of
>> > > > > sources. It enables us to get rid of the source
>> > > interfaces.
>> > > > However,
>> > > > > SDF
>> > > > > does not solve the lack of streaming sources in
>> > Python.
>> > > > >
>> > > > > The expansion PR (thanks btw!) solves the
>> > problem of
>> > > > > expanding/translating URNs known to an
>> > > ExpansionService. That
>> > > > is a more
>> > > > > programmatic way of replacing language-specific
>> > performs,
>> > > > instead of
>> > > > > relying on translators directly in the Runner.
>> > > > >
>> > > > > What is unsolved is the configuration of
>> > sources from
>> > > a foreign
>> > > > > environment. In my opinion this is the most
>> > pressing issue
>> > > > for Python
>> > > > > sources, because what is PubSubIO worth in
>> > Python if
>> > > you cannot
>> > > > > configure it?
>> > > > >
>> > > > > What about this:
>> > > > >
>> > > > > I think it is worth adding a JSON configuration
>> > option for
>> > > > all existing
>> > > > > Java sources. That way, we could easily
>> > configure them as
>> > > > part of the
>> > > > > expansion request (which would contain a JSON
>> > > configuration).
>> > > > I'll
>> > > > > probably fork a thread to discuss this in more
>> > detail,
>> > > but would
>> > > > > like to
>> > > > > hear your thoughts.
>> > > > >
>> > > > > -Max
>> > > > >
>> > > > > On 01.02.19 13:08, Robert Bradshaw wrote:
>> > > > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian
>> > Michels
>> > > > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
>> > > > > >
>> > > > > > Ah, I thought you meant native Flink
>> > transforms.
>> > > > > >
>> > > > > > Exactly! The translation code is already
>> > there.
>> > > The main
>> > > > > challenge
>> > > > > > is how to
>> > > > > > programmatically configure the BeamIO
>> from
>> > > Python. I
>> > > > suppose
>> > > > > that is
>> > > > > > also an
>> > > > > > unsolved problem for cross-language
>> > transforms
>> > > in general.
>> > > > > >
>> > > > > >
>> > > > > > This is what
>> > > https://github.com/apache/beam/pull/7316 does.
>> > > > > >
>> > > > > > For a particular source, one would want to
>> > define a
>> > > URN and
>> > > > > > corresponding payload, then (probably) a
>> > > CompositeTransform in
>> > > > > Python
>> > > > > > that takes the users arguments, packages
>> > them into the
>> > > > payload,
>> > > > > applies
>> > > > > > the ExternalTransform, and returns the
>> > results. How
>> > > to handle
>> > > > > arbitrary
>> > > > > > UDFs embedded in sources is still TBD.
>> > > > > >
>> > > > > > For Matthias' pipeline with PubSubIO we
>> > can build
>> > > > something
>> > > > > > specific, but for
>> > > > > > the general case there should be way to
>> > > initialize a
>> > > > Beam IO
>> > > > > via a
>> > > > > > configuration
>> > > > > > map provided by an external environment.
>> > > > > >
>> > > > > >
>> > > > > > I thought quite a bit about how we could
>> > represent
>> > > expansions
>> > > > > statically
>> > > > > > (e.g. have some kind of expansion template
>> that
>> > > could be
>> > > > used, at
>> > > > > least
>> > > > > > in many cases, as data without firing up a
>> > separate
>> > > > process. May be
>> > > > > > worth doing eventually, but we run into the
>> > same issues
>> > > > that were
>> > > > > > discussed at
>> > > > > >
>> > > >
>> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>> > > > > >
>> > > > > > If one is already using a portable runner
>> > like Flink,
>> > > > having the job
>> > > > > > service process automatically also serve up
>> an
>> > > expansion
>> > > > service for
>> > > > > > various URNs it knows and cares about is
>> > probably a
>> > > pretty
>> > > > low bar.
>> > > > > > Flink could serve up things it would rather
>> > get back
>> > > > untouched in a
>> > > > > > transform with a special flink runner urn.
>> > > > > >
>> > > > > > As Ahmet mentions, SDF is better solution. I
>> > hope
>> > > it's not
>> > > > that far
>> > > > > > away, but even once it comes we'll likely
>> > want the
>> > > above
>> > > > > framework to
>> > > > > > invoke the full suite of Java IOs even after
>> > they're
>> > > > running on SDF
>> > > > > > themselves.
>> > > > > >
>> > > > > > - Robert
>> > > > > >
>> > > > > > On 31.01.19 17:36, Thomas Weise wrote:
>> > > > > > > Exactly, that's what I had in mind.
>> > > > > > >
>> > > > > > > A Flink runner native transform would
>> > make
>> > > the existing
>> > > > > unbounded
>> > > > > > sources
>> > > > > > > available, similar to:
>> > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > >
>> > > > > > > On Thu, Jan 31, 2019 at 8:18 AM
>> > Maximilian
>> > > Michels
>> > > > > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
>> > > > > > >
>> > > > > > > Wouldn't it be even more useful
>> > for the
>> > > transition
>> > > > > period if
>> > > > > > we enabled Beam IO
>> > > > > > > to be used via Flink (like in the
>> > legacy
>> > > Flink
>> > > > Runner)? In
>> > > > > > this particular
>> > > > > > > example, Matthias wants to use
>> > PubSubIO,
>> > > which
>> > > > is not even
>> > > > > > available as a
>> > > > > > > native
>> > > > > > > Flink transform.
>> > > > > > >
>> > > > > > > On 31.01.19 16:21, Thomas Weise
>> > wrote:
>> > > > > > > > Until SDF is supported, we
>> > could also add
>> > > > Flink runner
>> > > > > > native transforms for
>> > > > > > > > selected unbounded sources
>> [1].
>> > > > > > > >
>> > > > > > > > That might be a reasonable
>> > option to
>> > > unblock
>> > > > users that
>> > > > > > want to try Python
>> > > > > > > > streaming on Flink.
>> > > > > > > >
>> > > > > > > > Thomas
>> > > > > > > >
>> > > > > > > > [1]
>> > > > > > > >
>> > > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>> > > > > > > >
>> > > > > > > >
>> > > > > > > > On Thu, Jan 31, 2019 at 6:51
>> AM
>> > > Maximilian
>> > > > Michels
>> > > > > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>> > > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
>> > > > > > > >
>> > > > > > > > > I have a hard time to
>> > imagine
>> > > how can
>> > > > we map
>> > > > > in a
>> > > > > > generic way
>> > > > > > > > RestrictionTrackers into
>> > the existing
>> > > > > > Bounded/UnboundedSource, so I would
>> > > > > > > > love to hear more about
>> > the details.
>> > > > > > > >
>> > > > > > > > Isn't it the other way
>> around?
>> > > The SDF is a
>> > > > > > generalization of
>> > > > > > > UnboundedSource.
>> > > > > > > > So we would wrap
>> > UnboundedSource
>> > > using
>> > > > SDF. I'm not
>> > > > > > saying it is
>> > > > > > > trivial, but
>> > > > > > > > SDF offers all the
>> > functionality that
>> > > > > UnboundedSource
>> > > > > > needs.
>> > > > > > > >
>> > > > > > > > For example, the
>> > > @GetInitialRestriction
>> > > > method
>> > > > > would
>> > > > > > call split on the
>> > > > > > > > UnboundedSource and the
>> > restriction
>> > > > trackers would
>> > > > > > then be used to
>> > > > > > > process the
>> > > > > > > > splits.
>> > > > > > > >
>> > > > > > > > On 31.01.19 15:16, Ismaël
>> > Mejía
>> > > wrote:
>> > > > > > > > >> Not necessarily. This
>> > would
>> > > be one
>> > > > way. Another
>> > > > > > way is build an SDF
>> > > > > > > > wrapper for
>> UnboundedSource.
>> > > Probably the
>> > > > > easier path
>> > > > > > for migration.
>> > > > > > > > >
>> > > > > > > > > That would be
>> fantastic, I
>> > > have heard
>> > > > about such
>> > > > > > wrapper multiple
>> > > > > > > > > times but so far there
>> > is not any
>> > > > realistic
>> > > > > > proposal. I have a hard
>> > > > > > > > > time to imagine how can
>> > we map
>> > > in a
>> > > > generic way
>> > > > > > RestrictionTrackers
>> > > > > > > > > into the existing
>> > > > Bounded/UnboundedSource, so I
>> > > > > > would love to hear
>> > > > > > > > > more about the details.
>> > > > > > > > >
>> > > > > > > > > On Thu, Jan 31, 2019 at
>> > 3:07 PM
>> > > > Maximilian
>> > > > > Michels
>> > > > > > <mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>> > > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
>> > > > > > > > >>
>> > > > > > > > >> > In addition to
>> have
>> > > support in
>> > > > the runners,
>> > > > > > this will require a
>> > > > > > > > >> > rewrite of
>> PubsubIO to
>> > > use the
>> > > > new SDF API.
>> > > > > > > > >>
>> > > > > > > > >> Not necessarily. This
>> > would
>> > > be one
>> > > > way. Another
>> > > > > > way is build an SDF
>> > > > > > > > wrapper for
>> > > > > > > > >> UnboundedSource.
>> > Probably the
>> > > easier
>> > > > path for
>> > > > > > migration.
>> > > > > > > > >>
>> > > > > > > > >> On 31.01.19 14:03,
>> Ismaël
>> > > Mejía wrote:
>> > > > > > > > >>>> Fortunately, there
>> > is already a
>> > > > pending
>> > > > > PR for
>> > > > > > cross-language
>> > > > > > > > pipelines which
>> > > > > > > > >>>> will allow us to use
>> > Java
>> > > IO like
>> > > > PubSub in
>> > > > > > Python jobs.
>> > > > > > > > >>>
>> > > > > > > > >>> In addition to have
>> > support
>> > > in the
>> > > > > runners, this
>> > > > > > will require a
>> > > > > > > > >>> rewrite of PubsubIO
>> > to use
>> > > the new
>> > > > SDF API.
>> > > > > > > > >>>
>> > > > > > > > >>> On Thu, Jan 31, 2019
>> > at 12:23 PM
>> > > > > Maximilian Michels
>> > > > > > > <mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
>> > > > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org>
>> > > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
>> > > > > > <mailto:mxm@apache.org
>> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
>> mxm@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>>
>> > > > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>
>> > > <mailto:mxm@apache.org <ma...@apache.org>
>> > <mailto:mxm@apache.org <ma...@apache.org>>>
>> > > > <mail
>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
Can you please add the link to the design document webpage.
Le jeu. 7 févr. 2019 à 19:59, Maximilian Michels <mx...@apache.org> a écrit :
> I've created an initial design document:
> https://s.apache.org/beam-cross-language-io
>
> It does not contain all the details but perhaps it's a good basis for a
> discussion on how we proceed.
>
> -Max
>
> On 06.02.19 19:49, Chamikara Jayalath wrote:
> >
> >
> > On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Thanks for your replies Robert and Cham.
> >
> > What I had in mind was a generic Wrapper that would easily allow
> users
> > to use IO from Java. Such wrapper could start as an experimental
> > feature
> > and then, through URN versioning, become stable eventually.
> >
> > UDFs are needed, though they are a special case. Most users
> (including
> > Matthias) just want to specify a few String options which do not
> > require
> > UDFs but something along the lines what I proposed here.
> >
> >
> > Sounds good let's start documenting/implementing the "easy" case and
> > think bit more regarding UDFs.
> >
> >
> >
> > Robert wrote:
> > > UDFs that are called from within an IO as part of its operation is
> > > still an open question.
> >
> > Exactly. How about we solve the easier case first, unblock users, and
> > then think more about solving the general case?
> >
> > Cham wrote:
> > > I'm happy to work with you to realize this.
> >
> > Would be great to exchange more ideas on this! I can compile the
> > current
> > ideas we have in a document and we move from there.
> >
> >
> > Great. Thanks.
> >
> >
> > Thanks,
> > Max
> >
> > On 05.02.19 17:56, Chamikara Jayalath wrote:
> > >
> > >
> > > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>> wrote:
> > >
> > > Good points Cham.
> > >
> > > JSON seemed like the most intuitive way to specify a
> > configuration map.
> > > We already use JSON in other places, e.g. to specify the
> > environment
> > > configuration. It is not necessarily a contradiction to have
> JSON
> > > inside
> > > Protobuf. From the perspective of IO authors, the
> > user-friendliness
> > > plays a role because they wouldn't have to deal with Protobuf.
> > >
> > >
> > > It's a good point that JSON will make this more user-friendly for
> IO
> > > transforms authors. Probably we should do a bit of
> > experimentation and
> > > keep this experimental in case we hit a performance snag.
> > >
> > >
> > > I agree that the configuration format is an implementation
> > detail that
> > > will be hidden to users via easy-to-use wrappers.
> > >
> > > Do we have to support UDFs for expanding existing IO? Users
> > would still
> > > be able to apply UDFs via ParDo on the IO output collections.
> > Generally
> > > speaking, I can see how for cross-language transforms UDF
> > support would
> > > be good. For example, a Combine implementation in Java, where
> the
> > > combine UDFs come from Python.
> > >
> > >
> > > I think we should try to support UDFs in the first version unless
> > > there's a major blocker that hinders realizing this. Many IO
> > transforms
> > > available today expect users to pass UDFs to realize various
> > features
> > > (for example, dynamic destinations for BigqueryIO and FileIO,
> > timestamp
> > > function for KafkaIO). I think without support for UDFs usability
> of
> > > cross-language transforms feature will be significantly limited.
> > >
> > > I'm happy to work with you to realize this.
> > >
> > > Thanks,
> > > Cham
> > >
> > >
> > > I suppose the question is, do we try to solve the general
> > case, or
> > > do we
> > > go with a simpler approach for enabling the use of existing
> > IO first?
> > > Lack of IO seems to be the most pressing issue for the
> > adoption of Beam
> > > Python. I imagine that a backwards-compatible incremental
> > support for
> > > cross-language transforms (IOs first, later other transforms)
> > would be
> > > possible.
> > >
> > > -Max
> > >
> > > On 05.02.19 03:07, Chamikara Jayalath wrote:
> > > >
> > > >
> > > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Yes, I imagine sources to implement a JsonConfigurable
> > > interface (e.g.
> > > > on their builders):
> > > >
> > > > JsonConfigurable {
> > > > // Either a json string or Map<String, String>
> > > > apply(String jsonConfig);
> > > > }
> > > >
> > > > In Python we would create this transform:
> > > >
> > > > URN: JsonConfiguredSource:v1
> > > > payload: {
> > > > environment: environment_id, // Java/Python/Go
> > > > resourceIdentifier: string, //
> > > "org.apache.beam.io.PubSubIO"
> > > > configuration: json config, // { "topic" :
> > > "my_pubsub_topic" }
> > > > }
> > > >
> > > >
> > > > Thanks Max, this is a great first step towards defining to
> > API for
> > > > cross-language transforms.
> > > > Is there a reason why you would want to use JSON instead
> > of a proto
> > > > here. I guess we'll be providing a more user friendly
> language
> > > wrapper
> > > > (for example, Python) for end-users here, so
> > > user-friendliness-wise, the
> > > > format we choose won't matter much (for pipeline authors).
> > > > If we don't support UDFs, performance difference will be
> > > negligible, but
> > > > UDFs might require a callback to original SDK (per-element
> > worst
> > > case).
> > > > So might make sense to choose the more efficient format.
> > > >
> > > > Also, probably we need to define a more expanded definition
> > > (proto/JSON)
> > > > to support UDFs. For example, a payload + a set of
> parameter
> > > definitions
> > > > so that the target SDK (for example, Java) can call back
> the
> > > original
> > > > SDK where the pipeline was authored in (for example,
> > Python) to
> > > resolve
> > > > UDFs at runtime.
> > > >
> > > > Thanks,
> > > > Cham
> > > >
> > > > That's more generic and could be used for other
> languages
> > > where we
> > > > might
> > > > have sources/sinks.
> > > >
> > > > > (FWIW, I was imagining PubSubIO already had a
> > translation into
> > > > BeamFnApi protos that fully specified it, and we use
> > that same
> > > > format to translate back out.)
> > > >
> > > > Not that I know of.
> > > >
> > > > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > > > Are you suggesting something akin to a generic
> > > > >
> > > > > urn: JsonConfiguredJavaSource
> > > > > payload: some json specifying which source and
> > which
> > > parameters
> > > > >
> > > > > which would expand to actually constructing and
> > applying
> > > that source?
> > > > >
> > > > > (FWIW, I was imagining PubSubIO already had a
> > translation into
> > > > BeamFnApi
> > > > > protos that fully specified it, and we use that
> > same format to
> > > > translate
> > > > > back out.)
> > > > >
> > > > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Recaping here:
> > > > >
> > > > > We all agree that SDF is the way to go for
> future
> > > > implementations of
> > > > > sources. It enables us to get rid of the source
> > > interfaces.
> > > > However,
> > > > > SDF
> > > > > does not solve the lack of streaming sources in
> > Python.
> > > > >
> > > > > The expansion PR (thanks btw!) solves the
> > problem of
> > > > > expanding/translating URNs known to an
> > > ExpansionService. That
> > > > is a more
> > > > > programmatic way of replacing language-specific
> > performs,
> > > > instead of
> > > > > relying on translators directly in the Runner.
> > > > >
> > > > > What is unsolved is the configuration of
> > sources from
> > > a foreign
> > > > > environment. In my opinion this is the most
> > pressing issue
> > > > for Python
> > > > > sources, because what is PubSubIO worth in
> > Python if
> > > you cannot
> > > > > configure it?
> > > > >
> > > > > What about this:
> > > > >
> > > > > I think it is worth adding a JSON configuration
> > option for
> > > > all existing
> > > > > Java sources. That way, we could easily
> > configure them as
> > > > part of the
> > > > > expansion request (which would contain a JSON
> > > configuration).
> > > > I'll
> > > > > probably fork a thread to discuss this in more
> > detail,
> > > but would
> > > > > like to
> > > > > hear your thoughts.
> > > > >
> > > > > -Max
> > > > >
> > > > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian
> > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > Ah, I thought you meant native Flink
> > transforms.
> > > > > >
> > > > > > Exactly! The translation code is already
> > there.
> > > The main
> > > > > challenge
> > > > > > is how to
> > > > > > programmatically configure the BeamIO
> from
> > > Python. I
> > > > suppose
> > > > > that is
> > > > > > also an
> > > > > > unsolved problem for cross-language
> > transforms
> > > in general.
> > > > > >
> > > > > >
> > > > > > This is what
> > > https://github.com/apache/beam/pull/7316 does.
> > > > > >
> > > > > > For a particular source, one would want to
> > define a
> > > URN and
> > > > > > corresponding payload, then (probably) a
> > > CompositeTransform in
> > > > > Python
> > > > > > that takes the users arguments, packages
> > them into the
> > > > payload,
> > > > > applies
> > > > > > the ExternalTransform, and returns the
> > results. How
> > > to handle
> > > > > arbitrary
> > > > > > UDFs embedded in sources is still TBD.
> > > > > >
> > > > > > For Matthias' pipeline with PubSubIO we
> > can build
> > > > something
> > > > > > specific, but for
> > > > > > the general case there should be way to
> > > initialize a
> > > > Beam IO
> > > > > via a
> > > > > > configuration
> > > > > > map provided by an external environment.
> > > > > >
> > > > > >
> > > > > > I thought quite a bit about how we could
> > represent
> > > expansions
> > > > > statically
> > > > > > (e.g. have some kind of expansion template
> that
> > > could be
> > > > used, at
> > > > > least
> > > > > > in many cases, as data without firing up a
> > separate
> > > > process. May be
> > > > > > worth doing eventually, but we run into the
> > same issues
> > > > that were
> > > > > > discussed at
> > > > > >
> > > >
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > > > >
> > > > > > If one is already using a portable runner
> > like Flink,
> > > > having the job
> > > > > > service process automatically also serve up
> an
> > > expansion
> > > > service for
> > > > > > various URNs it knows and cares about is
> > probably a
> > > pretty
> > > > low bar.
> > > > > > Flink could serve up things it would rather
> > get back
> > > > untouched in a
> > > > > > transform with a special flink runner urn.
> > > > > >
> > > > > > As Ahmet mentions, SDF is better solution. I
> > hope
> > > it's not
> > > > that far
> > > > > > away, but even once it comes we'll likely
> > want the
> > > above
> > > > > framework to
> > > > > > invoke the full suite of Java IOs even after
> > they're
> > > > running on SDF
> > > > > > themselves.
> > > > > >
> > > > > > - Robert
> > > > > >
> > > > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > > > Exactly, that's what I had in mind.
> > > > > > >
> > > > > > > A Flink runner native transform would
> > make
> > > the existing
> > > > > unbounded
> > > > > > sources
> > > > > > > available, similar to:
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 8:18 AM
> > Maximilian
> > > Michels
> > > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >
> > > > > > > Wouldn't it be even more useful
> > for the
> > > transition
> > > > > period if
> > > > > > we enabled Beam IO
> > > > > > > to be used via Flink (like in the
> > legacy
> > > Flink
> > > > Runner)? In
> > > > > > this particular
> > > > > > > example, Matthias wants to use
> > PubSubIO,
> > > which
> > > > is not even
> > > > > > available as a
> > > > > > > native
> > > > > > > Flink transform.
> > > > > > >
> > > > > > > On 31.01.19 16:21, Thomas Weise
> > wrote:
> > > > > > > > Until SDF is supported, we
> > could also add
> > > > Flink runner
> > > > > > native transforms for
> > > > > > > > selected unbounded sources [1].
> > > > > > > >
> > > > > > > > That might be a reasonable
> > option to
> > > unblock
> > > > users that
> > > > > > want to try Python
> > > > > > > > streaming on Flink.
> > > > > > > >
> > > > > > > > Thomas
> > > > > > > >
> > > > > > > > [1]
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > > >
> > > > > > > >
> > > > > > > > On Thu, Jan 31, 2019 at 6:51 AM
> > > Maximilian
> > > > Michels
> > > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >
> > > > > > > > > I have a hard time to
> > imagine
> > > how can
> > > > we map
> > > > > in a
> > > > > > generic way
> > > > > > > > RestrictionTrackers into
> > the existing
> > > > > > Bounded/UnboundedSource, so I would
> > > > > > > > love to hear more about
> > the details.
> > > > > > > >
> > > > > > > > Isn't it the other way
> around?
> > > The SDF is a
> > > > > > generalization of
> > > > > > > UnboundedSource.
> > > > > > > > So we would wrap
> > UnboundedSource
> > > using
> > > > SDF. I'm not
> > > > > > saying it is
> > > > > > > trivial, but
> > > > > > > > SDF offers all the
> > functionality that
> > > > > UnboundedSource
> > > > > > needs.
> > > > > > > >
> > > > > > > > For example, the
> > > @GetInitialRestriction
> > > > method
> > > > > would
> > > > > > call split on the
> > > > > > > > UnboundedSource and the
> > restriction
> > > > trackers would
> > > > > > then be used to
> > > > > > > process the
> > > > > > > > splits.
> > > > > > > >
> > > > > > > > On 31.01.19 15:16, Ismaël
> > Mejía
> > > wrote:
> > > > > > > > >> Not necessarily. This
> > would
> > > be one
> > > > way. Another
> > > > > > way is build an SDF
> > > > > > > > wrapper for
> UnboundedSource.
> > > Probably the
> > > > > easier path
> > > > > > for migration.
> > > > > > > > >
> > > > > > > > > That would be
> fantastic, I
> > > have heard
> > > > about such
> > > > > > wrapper multiple
> > > > > > > > > times but so far there
> > is not any
> > > > realistic
> > > > > > proposal. I have a hard
> > > > > > > > > time to imagine how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers
> > > > > > > > > into the existing
> > > > Bounded/UnboundedSource, so I
> > > > > > would love to hear
> > > > > > > > > more about the details.
> > > > > > > > >
> > > > > > > > > On Thu, Jan 31, 2019 at
> > 3:07 PM
> > > > Maximilian
> > > > > Michels
> > > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > > >>
> > > > > > > > >> > In addition to have
> > > support in
> > > > the runners,
> > > > > > this will require a
> > > > > > > > >> > rewrite of
> PubsubIO to
> > > use the
> > > > new SDF API.
> > > > > > > > >>
> > > > > > > > >> Not necessarily. This
> > would
> > > be one
> > > > way. Another
> > > > > > way is build an SDF
> > > > > > > > wrapper for
> > > > > > > > >> UnboundedSource.
> > Probably the
> > > easier
> > > > path for
> > > > > > migration.
> > > > > > > > >>
> > > > > > > > >> On 31.01.19 14:03,
> Ismaël
> > > Mejía wrote:
> > > > > > > > >>>> Fortunately, there
> > is already a
> > > > pending
> > > > > PR for
> > > > > > cross-language
> > > > > > > > pipelines which
> > > > > > > > >>>> will allow us to use
> > Java
> > > IO like
> > > > PubSub in
> > > > > > Python jobs.
> > > > > > > > >>>
> > > > > > > > >>> In addition to have
> > support
> > > in the
> > > > > runners, this
> > > > > > will require a
> > > > > > > > >>> rewrite of PubsubIO
> > to use
> > > the new
> > > > SDF API.
> > > > > > > > >>>
> > > > > > > > >>> On Thu, Jan 31, 2019
> > at 12:23 PM
> > > > > Maximilian Michels
> > > > > > > <mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mail
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
I've created an initial design document:
https://s.apache.org/beam-cross-language-io
It does not contain all the details but perhaps it's a good basis for a
discussion on how we proceed.
-Max
On 06.02.19 19:49, Chamikara Jayalath wrote:
>
>
> On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Thanks for your replies Robert and Cham.
>
> What I had in mind was a generic Wrapper that would easily allow users
> to use IO from Java. Such wrapper could start as an experimental
> feature
> and then, through URN versioning, become stable eventually.
>
> UDFs are needed, though they are a special case. Most users (including
> Matthias) just want to specify a few String options which do not
> require
> UDFs but something along the lines what I proposed here.
>
>
> Sounds good let's start documenting/implementing the "easy" case and
> think bit more regarding UDFs.
>
>
>
> Robert wrote:
> > UDFs that are called from within an IO as part of its operation is
> > still an open question.
>
> Exactly. How about we solve the easier case first, unblock users, and
> then think more about solving the general case?
>
> Cham wrote:
> > I'm happy to work with you to realize this.
>
> Would be great to exchange more ideas on this! I can compile the
> current
> ideas we have in a document and we move from there.
>
>
> Great. Thanks.
>
>
> Thanks,
> Max
>
> On 05.02.19 17:56, Chamikara Jayalath wrote:
> >
> >
> > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>> wrote:
> >
> > Good points Cham.
> >
> > JSON seemed like the most intuitive way to specify a
> configuration map.
> > We already use JSON in other places, e.g. to specify the
> environment
> > configuration. It is not necessarily a contradiction to have JSON
> > inside
> > Protobuf. From the perspective of IO authors, the
> user-friendliness
> > plays a role because they wouldn't have to deal with Protobuf.
> >
> >
> > It's a good point that JSON will make this more user-friendly for IO
> > transforms authors. Probably we should do a bit of
> experimentation and
> > keep this experimental in case we hit a performance snag.
> >
> >
> > I agree that the configuration format is an implementation
> detail that
> > will be hidden to users via easy-to-use wrappers.
> >
> > Do we have to support UDFs for expanding existing IO? Users
> would still
> > be able to apply UDFs via ParDo on the IO output collections.
> Generally
> > speaking, I can see how for cross-language transforms UDF
> support would
> > be good. For example, a Combine implementation in Java, where the
> > combine UDFs come from Python.
> >
> >
> > I think we should try to support UDFs in the first version unless
> > there's a major blocker that hinders realizing this. Many IO
> transforms
> > available today expect users to pass UDFs to realize various
> features
> > (for example, dynamic destinations for BigqueryIO and FileIO,
> timestamp
> > function for KafkaIO). I think without support for UDFs usability of
> > cross-language transforms feature will be significantly limited.
> >
> > I'm happy to work with you to realize this.
> >
> > Thanks,
> > Cham
> >
> >
> > I suppose the question is, do we try to solve the general
> case, or
> > do we
> > go with a simpler approach for enabling the use of existing
> IO first?
> > Lack of IO seems to be the most pressing issue for the
> adoption of Beam
> > Python. I imagine that a backwards-compatible incremental
> support for
> > cross-language transforms (IOs first, later other transforms)
> would be
> > possible.
> >
> > -Max
> >
> > On 05.02.19 03:07, Chamikara Jayalath wrote:
> > >
> > >
> > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Yes, I imagine sources to implement a JsonConfigurable
> > interface (e.g.
> > > on their builders):
> > >
> > > JsonConfigurable {
> > > // Either a json string or Map<String, String>
> > > apply(String jsonConfig);
> > > }
> > >
> > > In Python we would create this transform:
> > >
> > > URN: JsonConfiguredSource:v1
> > > payload: {
> > > environment: environment_id, // Java/Python/Go
> > > resourceIdentifier: string, //
> > "org.apache.beam.io.PubSubIO"
> > > configuration: json config, // { "topic" :
> > "my_pubsub_topic" }
> > > }
> > >
> > >
> > > Thanks Max, this is a great first step towards defining to
> API for
> > > cross-language transforms.
> > > Is there a reason why you would want to use JSON instead
> of a proto
> > > here. I guess we'll be providing a more user friendly language
> > wrapper
> > > (for example, Python) for end-users here, so
> > user-friendliness-wise, the
> > > format we choose won't matter much (for pipeline authors).
> > > If we don't support UDFs, performance difference will be
> > negligible, but
> > > UDFs might require a callback to original SDK (per-element
> worst
> > case).
> > > So might make sense to choose the more efficient format.
> > >
> > > Also, probably we need to define a more expanded definition
> > (proto/JSON)
> > > to support UDFs. For example, a payload + a set of parameter
> > definitions
> > > so that the target SDK (for example, Java) can call back the
> > original
> > > SDK where the pipeline was authored in (for example,
> Python) to
> > resolve
> > > UDFs at runtime.
> > >
> > > Thanks,
> > > Cham
> > >
> > > That's more generic and could be used for other languages
> > where we
> > > might
> > > have sources/sinks.
> > >
> > > > (FWIW, I was imagining PubSubIO already had a
> translation into
> > > BeamFnApi protos that fully specified it, and we use
> that same
> > > format to translate back out.)
> > >
> > > Not that I know of.
> > >
> > > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > > Are you suggesting something akin to a generic
> > > >
> > > > urn: JsonConfiguredJavaSource
> > > > payload: some json specifying which source and
> which
> > parameters
> > > >
> > > > which would expand to actually constructing and
> applying
> > that source?
> > > >
> > > > (FWIW, I was imagining PubSubIO already had a
> translation into
> > > BeamFnApi
> > > > protos that fully specified it, and we use that
> same format to
> > > translate
> > > > back out.)
> > > >
> > > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > Recaping here:
> > > >
> > > > We all agree that SDF is the way to go for future
> > > implementations of
> > > > sources. It enables us to get rid of the source
> > interfaces.
> > > However,
> > > > SDF
> > > > does not solve the lack of streaming sources in
> Python.
> > > >
> > > > The expansion PR (thanks btw!) solves the
> problem of
> > > > expanding/translating URNs known to an
> > ExpansionService. That
> > > is a more
> > > > programmatic way of replacing language-specific
> performs,
> > > instead of
> > > > relying on translators directly in the Runner.
> > > >
> > > > What is unsolved is the configuration of
> sources from
> > a foreign
> > > > environment. In my opinion this is the most
> pressing issue
> > > for Python
> > > > sources, because what is PubSubIO worth in
> Python if
> > you cannot
> > > > configure it?
> > > >
> > > > What about this:
> > > >
> > > > I think it is worth adding a JSON configuration
> option for
> > > all existing
> > > > Java sources. That way, we could easily
> configure them as
> > > part of the
> > > > expansion request (which would contain a JSON
> > configuration).
> > > I'll
> > > > probably fork a thread to discuss this in more
> detail,
> > but would
> > > > like to
> > > > hear your thoughts.
> > > >
> > > > -Max
> > > >
> > > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian
> Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > Ah, I thought you meant native Flink
> transforms.
> > > > >
> > > > > Exactly! The translation code is already
> there.
> > The main
> > > > challenge
> > > > > is how to
> > > > > programmatically configure the BeamIO from
> > Python. I
> > > suppose
> > > > that is
> > > > > also an
> > > > > unsolved problem for cross-language
> transforms
> > in general.
> > > > >
> > > > >
> > > > > This is what
> > https://github.com/apache/beam/pull/7316 does.
> > > > >
> > > > > For a particular source, one would want to
> define a
> > URN and
> > > > > corresponding payload, then (probably) a
> > CompositeTransform in
> > > > Python
> > > > > that takes the users arguments, packages
> them into the
> > > payload,
> > > > applies
> > > > > the ExternalTransform, and returns the
> results. How
> > to handle
> > > > arbitrary
> > > > > UDFs embedded in sources is still TBD.
> > > > >
> > > > > For Matthias' pipeline with PubSubIO we
> can build
> > > something
> > > > > specific, but for
> > > > > the general case there should be way to
> > initialize a
> > > Beam IO
> > > > via a
> > > > > configuration
> > > > > map provided by an external environment.
> > > > >
> > > > >
> > > > > I thought quite a bit about how we could
> represent
> > expansions
> > > > statically
> > > > > (e.g. have some kind of expansion template that
> > could be
> > > used, at
> > > > least
> > > > > in many cases, as data without firing up a
> separate
> > > process. May be
> > > > > worth doing eventually, but we run into the
> same issues
> > > that were
> > > > > discussed at
> > > > >
> > >
> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > > >
> > > > > If one is already using a portable runner
> like Flink,
> > > having the job
> > > > > service process automatically also serve up an
> > expansion
> > > service for
> > > > > various URNs it knows and cares about is
> probably a
> > pretty
> > > low bar.
> > > > > Flink could serve up things it would rather
> get back
> > > untouched in a
> > > > > transform with a special flink runner urn.
> > > > >
> > > > > As Ahmet mentions, SDF is better solution. I
> hope
> > it's not
> > > that far
> > > > > away, but even once it comes we'll likely
> want the
> > above
> > > > framework to
> > > > > invoke the full suite of Java IOs even after
> they're
> > > running on SDF
> > > > > themselves.
> > > > >
> > > > > - Robert
> > > > >
> > > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > > Exactly, that's what I had in mind.
> > > > > >
> > > > > > A Flink runner native transform would
> make
> > the existing
> > > > unbounded
> > > > > sources
> > > > > > available, similar to:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 8:18 AM
> Maximilian
> > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > >
> > > > > > Wouldn't it be even more useful
> for the
> > transition
> > > > period if
> > > > > we enabled Beam IO
> > > > > > to be used via Flink (like in the
> legacy
> > Flink
> > > Runner)? In
> > > > > this particular
> > > > > > example, Matthias wants to use
> PubSubIO,
> > which
> > > is not even
> > > > > available as a
> > > > > > native
> > > > > > Flink transform.
> > > > > >
> > > > > > On 31.01.19 16:21, Thomas Weise
> wrote:
> > > > > > > Until SDF is supported, we
> could also add
> > > Flink runner
> > > > > native transforms for
> > > > > > > selected unbounded sources [1].
> > > > > > >
> > > > > > > That might be a reasonable
> option to
> > unblock
> > > users that
> > > > > want to try Python
> > > > > > > streaming on Flink.
> > > > > > >
> > > > > > > Thomas
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 6:51 AM
> > Maximilian
> > > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > >
> > > > > > > > I have a hard time to
> imagine
> > how can
> > > we map
> > > > in a
> > > > > generic way
> > > > > > > RestrictionTrackers into
> the existing
> > > > > Bounded/UnboundedSource, so I would
> > > > > > > love to hear more about
> the details.
> > > > > > >
> > > > > > > Isn't it the other way around?
> > The SDF is a
> > > > > generalization of
> > > > > > UnboundedSource.
> > > > > > > So we would wrap
> UnboundedSource
> > using
> > > SDF. I'm not
> > > > > saying it is
> > > > > > trivial, but
> > > > > > > SDF offers all the
> functionality that
> > > > UnboundedSource
> > > > > needs.
> > > > > > >
> > > > > > > For example, the
> > @GetInitialRestriction
> > > method
> > > > would
> > > > > call split on the
> > > > > > > UnboundedSource and the
> restriction
> > > trackers would
> > > > > then be used to
> > > > > > process the
> > > > > > > splits.
> > > > > > >
> > > > > > > On 31.01.19 15:16, Ismaël
> Mejía
> > wrote:
> > > > > > > >> Not necessarily. This
> would
> > be one
> > > way. Another
> > > > > way is build an SDF
> > > > > > > wrapper for UnboundedSource.
> > Probably the
> > > > easier path
> > > > > for migration.
> > > > > > > >
> > > > > > > > That would be fantastic, I
> > have heard
> > > about such
> > > > > wrapper multiple
> > > > > > > > times but so far there
> is not any
> > > realistic
> > > > > proposal. I have a hard
> > > > > > > > time to imagine how can
> we map
> > in a
> > > generic way
> > > > > RestrictionTrackers
> > > > > > > > into the existing
> > > Bounded/UnboundedSource, so I
> > > > > would love to hear
> > > > > > > > more about the details.
> > > > > > > >
> > > > > > > > On Thu, Jan 31, 2019 at
> 3:07 PM
> > > Maximilian
> > > > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >>
> > > > > > > >> > In addition to have
> > support in
> > > the runners,
> > > > > this will require a
> > > > > > > >> > rewrite of PubsubIO to
> > use the
> > > new SDF API.
> > > > > > > >>
> > > > > > > >> Not necessarily. This
> would
> > be one
> > > way. Another
> > > > > way is build an SDF
> > > > > > > wrapper for
> > > > > > > >> UnboundedSource.
> Probably the
> > easier
> > > path for
> > > > > migration.
> > > > > > > >>
> > > > > > > >> On 31.01.19 14:03, Ismaël
> > Mejía wrote:
> > > > > > > >>>> Fortunately, there
> is already a
> > > pending
> > > > PR for
> > > > > cross-language
> > > > > > > pipelines which
> > > > > > > >>>> will allow us to use
> Java
> > IO like
> > > PubSub in
> > > > > Python jobs.
> > > > > > > >>>
> > > > > > > >>> In addition to have
> support
> > in the
> > > > runners, this
> > > > > will require a
> > > > > > > >>> rewrite of PubsubIO
> to use
> > the new
> > > SDF API.
> > > > > > > >>>
> > > > > > > >>> On Thu, Jan 31, 2019
> at 12:23 PM
> > > > Maximilian Michels
> > > > > > <mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>
> > > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>> Hi Matthias,
> > > > > > > >>>>
> > > > > > > >>>> This is already
> reflected
> > in the
> > > > compatibility
> > > > > matrix, if you look
> > > > > > > under SDF.
> > > > > > > >>>> There is no
> UnboundedSource
> > > interface for
> > > > > portable pipelines.
> > > > > > That's a
> > > > > > > legacy
> > > > > > > >>>> abstraction that will be
> > replaced
> > > with SDF.
> > > > > > > >>>>
> > > > > > > >>>> Fortunately, there
> is already a
> > > pending
> > > > PR for
> > > > > cross-language
> > > > > > > pipelines which
> > > > > > > >>>> will allow us to use
> Java
> > IO like
> > > PubSub in
> > > > > Python jobs.
> > > > > > > >>>>
> > > > > > > >>>> Thanks,
> > > > > > > >>>> Max
> > > > > > > >>>>
> > > > > > > >>>> On 31.01.19 12:06,
> Matthias
> > > Baetens wrote:
> > > > > > > >>>>> Hey Ankur,
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks for the
> swift reply.
> > > Should I change
> > > > > this in the
> > > > > > capability matrix
> > > > > > > >>>>>
> > > > >
> > >
> <https://s.apache.org/apache-beam-portability-support-table>
> > > > then?
> > > > > > > >>>>>
> > > > > > > >>>>> Many thanks.
> > > > > > > >>>>> Best,
> > > > > > > >>>>> Matthias
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, 31 Jan 2019
> at 09:31,
> > > Ankur Goenka
> > > > > <goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>
> > > > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>>
> > > > > > > >>>>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>
> > > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>> Hi Matthias,
> > > > > > > >>>>>
> > > > > > > >>>>> Unfortunately,
> > unbounded reads
> > > > including
> > > > > pubsub are not yet
> > > > > > > supported for
> > > > > > > >>>>> portable runners.
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks,
> > > > > > > >>>>> Ankur
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Jan
> 31, 2019
> > at 2:44 PM
> > > > Matthias
> > > > > Baetens
> > > > > > > <baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>>
> > > > > > > >>>>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>
> > > > > > >
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>> Hi everyone,
> > > > > > > >>>>>
> > > > > > > >>>>> Last few
> days I
> > have been
> > > > trying to
> > > > > run a streaming
> > > > > > > pipeline (code on
> > > > > > > >>>>> Github
> > > > >
> <https://github.com/matthiasa4/beam-demo>) on a
> > > > > > > Flink Runner.
> > > > > > > >>>>>
> > > > > > > >>>>> I am
> running a Flink
> > > cluster
> > > > locally
> > > > > (v1.5.6
> > > > > > > >>>>>
> > > > > <https://flink.apache.org/downloads.html>)
> > > > > > > >>>>> I have
> built the SDK
> > > Harness
> > > > > Container: /./gradlew
> > > > > > > >>>>>
> > > > :beam-sdks-python-container:docker/
> > > > > > > >>>>> and
> started the
> > JobServer:
> > > > /./gradlew
> > > > > > > >>>>>
> > > > >
> :beam-runners-flink_2.11-job-server:runShadow
> > > > > > > >>>>>
> > > -PflinkMasterUrl=localhost:8081./
> > > > > > > >>>>>
> > > > > > > >>>>> I run my
> > pipeline with:
> > > > /env/bin/python
> > > > > > streaming_pipeline.py
> > > > > > > >>>>>
> > --runner=PortableRunner
> > > > > --job_endpoint=localhost:8099
> > > > > > > --output xxx
> > > > > > > >>>>>
> > --input_subscription xxx
> > > > > --output_subscription xxx/
> > > > > > > >>>>> /
> > > > > > > >>>>> /
> > > > > > > >>>>> All this
> is running
> > > inside a
> > > > Ubuntu
> > > > > (Bionic) in a
> > > > > > Virtualbox.
> > > > > > > >>>>>
> > > > > > > >>>>> The job
> submits
> > fine, but
> > > > > unfortunately fails after
> > > > > > a few
> > > > > > > seconds with
> > > > > > > >>>>> the error
> attached.
> > > > > > > >>>>>
> > > > > > > >>>>> Anything I am
> > missing
> > > or doing
> > > > wrong?
> > > > > > > >>>>>
> > > > > > > >>>>> Many thanks.
> > > > > > > >>>>> Best,
> > > > > > > >>>>> Matthias
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Chamikara Jayalath <ch...@google.com>.
On Wed, Feb 6, 2019 at 8:38 AM Maximilian Michels <mx...@apache.org> wrote:
> Thanks for your replies Robert and Cham.
>
> What I had in mind was a generic Wrapper that would easily allow users
> to use IO from Java. Such wrapper could start as an experimental feature
> and then, through URN versioning, become stable eventually.
>
> UDFs are needed, though they are a special case. Most users (including
> Matthias) just want to specify a few String options which do not require
> UDFs but something along the lines what I proposed here.
>
Sounds good let's start documenting/implementing the "easy" case and think
bit more regarding UDFs.
> Robert wrote:
> > UDFs that are called from within an IO as part of its operation is
> > still an open question.
>
> Exactly. How about we solve the easier case first, unblock users, and
> then think more about solving the general case?
>
> Cham wrote:
> > I'm happy to work with you to realize this.
>
> Would be great to exchange more ideas on this! I can compile the current
> ideas we have in a document and we move from there.
>
Great. Thanks.
>
> Thanks,
> Max
>
> On 05.02.19 17:56, Chamikara Jayalath wrote:
> >
> >
> > On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org> wrote:
> >
> > Good points Cham.
> >
> > JSON seemed like the most intuitive way to specify a configuration
> map.
> > We already use JSON in other places, e.g. to specify the environment
> > configuration. It is not necessarily a contradiction to have JSON
> > inside
> > Protobuf. From the perspective of IO authors, the user-friendliness
> > plays a role because they wouldn't have to deal with Protobuf.
> >
> >
> > It's a good point that JSON will make this more user-friendly for IO
> > transforms authors. Probably we should do a bit of experimentation and
> > keep this experimental in case we hit a performance snag.
> >
> >
> > I agree that the configuration format is an implementation detail
> that
> > will be hidden to users via easy-to-use wrappers.
> >
> > Do we have to support UDFs for expanding existing IO? Users would
> still
> > be able to apply UDFs via ParDo on the IO output collections.
> Generally
> > speaking, I can see how for cross-language transforms UDF support
> would
> > be good. For example, a Combine implementation in Java, where the
> > combine UDFs come from Python.
> >
> >
> > I think we should try to support UDFs in the first version unless
> > there's a major blocker that hinders realizing this. Many IO transforms
> > available today expect users to pass UDFs to realize various features
> > (for example, dynamic destinations for BigqueryIO and FileIO, timestamp
> > function for KafkaIO). I think without support for UDFs usability of
> > cross-language transforms feature will be significantly limited.
> >
> > I'm happy to work with you to realize this.
> >
> > Thanks,
> > Cham
> >
> >
> > I suppose the question is, do we try to solve the general case, or
> > do we
> > go with a simpler approach for enabling the use of existing IO first?
> > Lack of IO seems to be the most pressing issue for the adoption of
> Beam
> > Python. I imagine that a backwards-compatible incremental support for
> > cross-language transforms (IOs first, later other transforms) would
> be
> > possible.
> >
> > -Max
> >
> > On 05.02.19 03:07, Chamikara Jayalath wrote:
> > >
> > >
> > > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Yes, I imagine sources to implement a JsonConfigurable
> > interface (e.g.
> > > on their builders):
> > >
> > > JsonConfigurable {
> > > // Either a json string or Map<String, String>
> > > apply(String jsonConfig);
> > > }
> > >
> > > In Python we would create this transform:
> > >
> > > URN: JsonConfiguredSource:v1
> > > payload: {
> > > environment: environment_id, // Java/Python/Go
> > > resourceIdentifier: string, //
> > "org.apache.beam.io.PubSubIO"
> > > configuration: json config, // { "topic" :
> > "my_pubsub_topic" }
> > > }
> > >
> > >
> > > Thanks Max, this is a great first step towards defining to API for
> > > cross-language transforms.
> > > Is there a reason why you would want to use JSON instead of a
> proto
> > > here. I guess we'll be providing a more user friendly language
> > wrapper
> > > (for example, Python) for end-users here, so
> > user-friendliness-wise, the
> > > format we choose won't matter much (for pipeline authors).
> > > If we don't support UDFs, performance difference will be
> > negligible, but
> > > UDFs might require a callback to original SDK (per-element worst
> > case).
> > > So might make sense to choose the more efficient format.
> > >
> > > Also, probably we need to define a more expanded definition
> > (proto/JSON)
> > > to support UDFs. For example, a payload + a set of parameter
> > definitions
> > > so that the target SDK (for example, Java) can call back the
> > original
> > > SDK where the pipeline was authored in (for example, Python) to
> > resolve
> > > UDFs at runtime.
> > >
> > > Thanks,
> > > Cham
> > >
> > > That's more generic and could be used for other languages
> > where we
> > > might
> > > have sources/sinks.
> > >
> > > > (FWIW, I was imagining PubSubIO already had a translation
> into
> > > BeamFnApi protos that fully specified it, and we use that same
> > > format to translate back out.)
> > >
> > > Not that I know of.
> > >
> > > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > > Are you suggesting something akin to a generic
> > > >
> > > > urn: JsonConfiguredJavaSource
> > > > payload: some json specifying which source and which
> > parameters
> > > >
> > > > which would expand to actually constructing and applying
> > that source?
> > > >
> > > > (FWIW, I was imagining PubSubIO already had a translation
> into
> > > BeamFnApi
> > > > protos that fully specified it, and we use that same
> format to
> > > translate
> > > > back out.)
> > > >
> > > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Recaping here:
> > > >
> > > > We all agree that SDF is the way to go for future
> > > implementations of
> > > > sources. It enables us to get rid of the source
> > interfaces.
> > > However,
> > > > SDF
> > > > does not solve the lack of streaming sources in Python.
> > > >
> > > > The expansion PR (thanks btw!) solves the problem of
> > > > expanding/translating URNs known to an
> > ExpansionService. That
> > > is a more
> > > > programmatic way of replacing language-specific
> performs,
> > > instead of
> > > > relying on translators directly in the Runner.
> > > >
> > > > What is unsolved is the configuration of sources from
> > a foreign
> > > > environment. In my opinion this is the most pressing
> issue
> > > for Python
> > > > sources, because what is PubSubIO worth in Python if
> > you cannot
> > > > configure it?
> > > >
> > > > What about this:
> > > >
> > > > I think it is worth adding a JSON configuration option
> for
> > > all existing
> > > > Java sources. That way, we could easily configure them
> as
> > > part of the
> > > > expansion request (which would contain a JSON
> > configuration).
> > > I'll
> > > > probably fork a thread to discuss this in more detail,
> > but would
> > > > like to
> > > > hear your thoughts.
> > > >
> > > > -Max
> > > >
> > > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Ah, I thought you meant native Flink transforms.
> > > > >
> > > > > Exactly! The translation code is already there.
> > The main
> > > > challenge
> > > > > is how to
> > > > > programmatically configure the BeamIO from
> > Python. I
> > > suppose
> > > > that is
> > > > > also an
> > > > > unsolved problem for cross-language transforms
> > in general.
> > > > >
> > > > >
> > > > > This is what
> > https://github.com/apache/beam/pull/7316 does.
> > > > >
> > > > > For a particular source, one would want to define a
> > URN and
> > > > > corresponding payload, then (probably) a
> > CompositeTransform in
> > > > Python
> > > > > that takes the users arguments, packages them into
> the
> > > payload,
> > > > applies
> > > > > the ExternalTransform, and returns the results. How
> > to handle
> > > > arbitrary
> > > > > UDFs embedded in sources is still TBD.
> > > > >
> > > > > For Matthias' pipeline with PubSubIO we can
> build
> > > something
> > > > > specific, but for
> > > > > the general case there should be way to
> > initialize a
> > > Beam IO
> > > > via a
> > > > > configuration
> > > > > map provided by an external environment.
> > > > >
> > > > >
> > > > > I thought quite a bit about how we could represent
> > expansions
> > > > statically
> > > > > (e.g. have some kind of expansion template that
> > could be
> > > used, at
> > > > least
> > > > > in many cases, as data without firing up a separate
> > > process. May be
> > > > > worth doing eventually, but we run into the same
> issues
> > > that were
> > > > > discussed at
> > > > >
> > > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > > >
> > > > > If one is already using a portable runner like
> Flink,
> > > having the job
> > > > > service process automatically also serve up an
> > expansion
> > > service for
> > > > > various URNs it knows and cares about is probably a
> > pretty
> > > low bar.
> > > > > Flink could serve up things it would rather get back
> > > untouched in a
> > > > > transform with a special flink runner urn.
> > > > >
> > > > > As Ahmet mentions, SDF is better solution. I hope
> > it's not
> > > that far
> > > > > away, but even once it comes we'll likely want the
> > above
> > > > framework to
> > > > > invoke the full suite of Java IOs even after they're
> > > running on SDF
> > > > > themselves.
> > > > >
> > > > > - Robert
> > > > >
> > > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > > Exactly, that's what I had in mind.
> > > > > >
> > > > > > A Flink runner native transform would make
> > the existing
> > > > unbounded
> > > > > sources
> > > > > > available, similar to:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian
> > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > Wouldn't it be even more useful for the
> > transition
> > > > period if
> > > > > we enabled Beam IO
> > > > > > to be used via Flink (like in the legacy
> > Flink
> > > Runner)? In
> > > > > this particular
> > > > > > example, Matthias wants to use PubSubIO,
> > which
> > > is not even
> > > > > available as a
> > > > > > native
> > > > > > Flink transform.
> > > > > >
> > > > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > > > Until SDF is supported, we could also
> add
> > > Flink runner
> > > > > native transforms for
> > > > > > > selected unbounded sources [1].
> > > > > > >
> > > > > > > That might be a reasonable option to
> > unblock
> > > users that
> > > > > want to try Python
> > > > > > > streaming on Flink.
> > > > > > >
> > > > > > > Thomas
> > > > > > >
> > > > > > > [1]
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 6:51 AM
> > Maximilian
> > > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >
> > > > > > > > I have a hard time to imagine
> > how can
> > > we map
> > > > in a
> > > > > generic way
> > > > > > > RestrictionTrackers into the
> existing
> > > > > Bounded/UnboundedSource, so I would
> > > > > > > love to hear more about the
> details.
> > > > > > >
> > > > > > > Isn't it the other way around?
> > The SDF is a
> > > > > generalization of
> > > > > > UnboundedSource.
> > > > > > > So we would wrap UnboundedSource
> > using
> > > SDF. I'm not
> > > > > saying it is
> > > > > > trivial, but
> > > > > > > SDF offers all the functionality
> that
> > > > UnboundedSource
> > > > > needs.
> > > > > > >
> > > > > > > For example, the
> > @GetInitialRestriction
> > > method
> > > > would
> > > > > call split on the
> > > > > > > UnboundedSource and the
> restriction
> > > trackers would
> > > > > then be used to
> > > > > > process the
> > > > > > > splits.
> > > > > > >
> > > > > > > On 31.01.19 15:16, Ismaël Mejía
> > wrote:
> > > > > > > >> Not necessarily. This would
> > be one
> > > way. Another
> > > > > way is build an SDF
> > > > > > > wrapper for UnboundedSource.
> > Probably the
> > > > easier path
> > > > > for migration.
> > > > > > > >
> > > > > > > > That would be fantastic, I
> > have heard
> > > about such
> > > > > wrapper multiple
> > > > > > > > times but so far there is not
> any
> > > realistic
> > > > > proposal. I have a hard
> > > > > > > > time to imagine how can we map
> > in a
> > > generic way
> > > > > RestrictionTrackers
> > > > > > > > into the existing
> > > Bounded/UnboundedSource, so I
> > > > > would love to hear
> > > > > > > > more about the details.
> > > > > > > >
> > > > > > > > On Thu, Jan 31, 2019 at 3:07 PM
> > > Maximilian
> > > > Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > > >>
> > > > > > > >> > In addition to have
> > support in
> > > the runners,
> > > > > this will require a
> > > > > > > >> > rewrite of PubsubIO to
> > use the
> > > new SDF API.
> > > > > > > >>
> > > > > > > >> Not necessarily. This would
> > be one
> > > way. Another
> > > > > way is build an SDF
> > > > > > > wrapper for
> > > > > > > >> UnboundedSource. Probably the
> > easier
> > > path for
> > > > > migration.
> > > > > > > >>
> > > > > > > >> On 31.01.19 14:03, Ismaël
> > Mejía wrote:
> > > > > > > >>>> Fortunately, there is
> already a
> > > pending
> > > > PR for
> > > > > cross-language
> > > > > > > pipelines which
> > > > > > > >>>> will allow us to use Java
> > IO like
> > > PubSub in
> > > > > Python jobs.
> > > > > > > >>>
> > > > > > > >>> In addition to have support
> > in the
> > > > runners, this
> > > > > will require a
> > > > > > > >>> rewrite of PubsubIO to use
> > the new
> > > SDF API.
> > > > > > > >>>
> > > > > > > >>> On Thu, Jan 31, 2019 at
> 12:23 PM
> > > > Maximilian Michels
> > > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > > >>>>
> > > > > > > >>>> Hi Matthias,
> > > > > > > >>>>
> > > > > > > >>>> This is already reflected
> > in the
> > > > compatibility
> > > > > matrix, if you look
> > > > > > > under SDF.
> > > > > > > >>>> There is no UnboundedSource
> > > interface for
> > > > > portable pipelines.
> > > > > > That's a
> > > > > > > legacy
> > > > > > > >>>> abstraction that will be
> > replaced
> > > with SDF.
> > > > > > > >>>>
> > > > > > > >>>> Fortunately, there is
> already a
> > > pending
> > > > PR for
> > > > > cross-language
> > > > > > > pipelines which
> > > > > > > >>>> will allow us to use Java
> > IO like
> > > PubSub in
> > > > > Python jobs.
> > > > > > > >>>>
> > > > > > > >>>> Thanks,
> > > > > > > >>>> Max
> > > > > > > >>>>
> > > > > > > >>>> On 31.01.19 12:06, Matthias
> > > Baetens wrote:
> > > > > > > >>>>> Hey Ankur,
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks for the swift reply.
> > > Should I change
> > > > > this in the
> > > > > > capability matrix
> > > > > > > >>>>>
> > > > >
> > > <https://s.apache.org/apache-beam-portability-support-table
> >
> > > > then?
> > > > > > > >>>>>
> > > > > > > >>>>> Many thanks.
> > > > > > > >>>>> Best,
> > > > > > > >>>>> Matthias
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, 31 Jan 2019 at
> 09:31,
> > > Ankur Goenka
> > > > > <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > > <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > > <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>>
> > > > > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>> Hi Matthias,
> > > > > > > >>>>>
> > > > > > > >>>>> Unfortunately,
> > unbounded reads
> > > > including
> > > > > pubsub are not yet
> > > > > > > supported for
> > > > > > > >>>>> portable runners.
> > > > > > > >>>>>
> > > > > > > >>>>> Thanks,
> > > > > > > >>>>> Ankur
> > > > > > > >>>>>
> > > > > > > >>>>> On Thu, Jan 31, 2019
> > at 2:44 PM
> > > > Matthias
> > > > > Baetens
> > > > > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>>
> > > > > > > >>>>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>
> > > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>>>> wrote:
> > > > > > > >>>>>
> > > > > > > >>>>> Hi everyone,
> > > > > > > >>>>>
> > > > > > > >>>>> Last few days I
> > have been
> > > > trying to
> > > > > run a streaming
> > > > > > > pipeline (code on
> > > > > > > >>>>> Github
> > > > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > > > Flink Runner.
> > > > > > > >>>>>
> > > > > > > >>>>> I am running a
> Flink
> > > cluster
> > > > locally
> > > > > (v1.5.6
> > > > > > > >>>>>
> > > > > <https://flink.apache.org/downloads.html>)
> > > > > > > >>>>> I have built the
> SDK
> > > Harness
> > > > > Container: /./gradlew
> > > > > > > >>>>>
> > > > :beam-sdks-python-container:docker/
> > > > > > > >>>>> and started the
> > JobServer:
> > > > /./gradlew
> > > > > > > >>>>>
> > > > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > > > >>>>>
> > > -PflinkMasterUrl=localhost:8081./
> > > > > > > >>>>>
> > > > > > > >>>>> I run my
> > pipeline with:
> > > > /env/bin/python
> > > > > > streaming_pipeline.py
> > > > > > > >>>>>
> > --runner=PortableRunner
> > > > > --job_endpoint=localhost:8099
> > > > > > > --output xxx
> > > > > > > >>>>>
> > --input_subscription xxx
> > > > > --output_subscription xxx/
> > > > > > > >>>>> /
> > > > > > > >>>>> /
> > > > > > > >>>>> All this is
> running
> > > inside a
> > > > Ubuntu
> > > > > (Bionic) in a
> > > > > > Virtualbox.
> > > > > > > >>>>>
> > > > > > > >>>>> The job submits
> > fine, but
> > > > > unfortunately fails after
> > > > > > a few
> > > > > > > seconds with
> > > > > > > >>>>> the error
> attached.
> > > > > > > >>>>>
> > > > > > > >>>>> Anything I am
> > missing
> > > or doing
> > > > wrong?
> > > > > > > >>>>>
> > > > > > > >>>>> Many thanks.
> > > > > > > >>>>> Best,
> > > > > > > >>>>> Matthias
> > > > > > > >>>>>
> > > > > > > >>>>>
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Thanks for your replies Robert and Cham.
What I had in mind was a generic Wrapper that would easily allow users
to use IO from Java. Such wrapper could start as an experimental feature
and then, through URN versioning, become stable eventually.
UDFs are needed, though they are a special case. Most users (including
Matthias) just want to specify a few String options which do not require
UDFs but something along the lines what I proposed here.
Robert wrote:
> UDFs that are called from within an IO as part of its operation is
> still an open question.
Exactly. How about we solve the easier case first, unblock users, and
then think more about solving the general case?
Cham wrote:
> I'm happy to work with you to realize this.
Would be great to exchange more ideas on this! I can compile the current
ideas we have in a document and we move from there.
Thanks,
Max
On 05.02.19 17:56, Chamikara Jayalath wrote:
>
>
> On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org> wrote:
>
> Good points Cham.
>
> JSON seemed like the most intuitive way to specify a configuration map.
> We already use JSON in other places, e.g. to specify the environment
> configuration. It is not necessarily a contradiction to have JSON
> inside
> Protobuf. From the perspective of IO authors, the user-friendliness
> plays a role because they wouldn't have to deal with Protobuf.
>
>
> It's a good point that JSON will make this more user-friendly for IO
> transforms authors. Probably we should do a bit of experimentation and
> keep this experimental in case we hit a performance snag.
>
>
> I agree that the configuration format is an implementation detail that
> will be hidden to users via easy-to-use wrappers.
>
> Do we have to support UDFs for expanding existing IO? Users would still
> be able to apply UDFs via ParDo on the IO output collections. Generally
> speaking, I can see how for cross-language transforms UDF support would
> be good. For example, a Combine implementation in Java, where the
> combine UDFs come from Python.
>
>
> I think we should try to support UDFs in the first version unless
> there's a major blocker that hinders realizing this. Many IO transforms
> available today expect users to pass UDFs to realize various features
> (for example, dynamic destinations for BigqueryIO and FileIO, timestamp
> function for KafkaIO). I think without support for UDFs usability of
> cross-language transforms feature will be significantly limited.
>
> I'm happy to work with you to realize this.
>
> Thanks,
> Cham
>
>
> I suppose the question is, do we try to solve the general case, or
> do we
> go with a simpler approach for enabling the use of existing IO first?
> Lack of IO seems to be the most pressing issue for the adoption of Beam
> Python. I imagine that a backwards-compatible incremental support for
> cross-language transforms (IOs first, later other transforms) would be
> possible.
>
> -Max
>
> On 05.02.19 03:07, Chamikara Jayalath wrote:
> >
> >
> > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Yes, I imagine sources to implement a JsonConfigurable
> interface (e.g.
> > on their builders):
> >
> > JsonConfigurable {
> > // Either a json string or Map<String, String>
> > apply(String jsonConfig);
> > }
> >
> > In Python we would create this transform:
> >
> > URN: JsonConfiguredSource:v1
> > payload: {
> > environment: environment_id, // Java/Python/Go
> > resourceIdentifier: string, //
> "org.apache.beam.io.PubSubIO"
> > configuration: json config, // { "topic" :
> "my_pubsub_topic" }
> > }
> >
> >
> > Thanks Max, this is a great first step towards defining to API for
> > cross-language transforms.
> > Is there a reason why you would want to use JSON instead of a proto
> > here. I guess we'll be providing a more user friendly language
> wrapper
> > (for example, Python) for end-users here, so
> user-friendliness-wise, the
> > format we choose won't matter much (for pipeline authors).
> > If we don't support UDFs, performance difference will be
> negligible, but
> > UDFs might require a callback to original SDK (per-element worst
> case).
> > So might make sense to choose the more efficient format.
> >
> > Also, probably we need to define a more expanded definition
> (proto/JSON)
> > to support UDFs. For example, a payload + a set of parameter
> definitions
> > so that the target SDK (for example, Java) can call back the
> original
> > SDK where the pipeline was authored in (for example, Python) to
> resolve
> > UDFs at runtime.
> >
> > Thanks,
> > Cham
> >
> > That's more generic and could be used for other languages
> where we
> > might
> > have sources/sinks.
> >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi protos that fully specified it, and we use that same
> > format to translate back out.)
> >
> > Not that I know of.
> >
> > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > Are you suggesting something akin to a generic
> > >
> > > urn: JsonConfiguredJavaSource
> > > payload: some json specifying which source and which
> parameters
> > >
> > > which would expand to actually constructing and applying
> that source?
> > >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi
> > > protos that fully specified it, and we use that same format to
> > translate
> > > back out.)
> > >
> > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Recaping here:
> > >
> > > We all agree that SDF is the way to go for future
> > implementations of
> > > sources. It enables us to get rid of the source
> interfaces.
> > However,
> > > SDF
> > > does not solve the lack of streaming sources in Python.
> > >
> > > The expansion PR (thanks btw!) solves the problem of
> > > expanding/translating URNs known to an
> ExpansionService. That
> > is a more
> > > programmatic way of replacing language-specific performs,
> > instead of
> > > relying on translators directly in the Runner.
> > >
> > > What is unsolved is the configuration of sources from
> a foreign
> > > environment. In my opinion this is the most pressing issue
> > for Python
> > > sources, because what is PubSubIO worth in Python if
> you cannot
> > > configure it?
> > >
> > > What about this:
> > >
> > > I think it is worth adding a JSON configuration option for
> > all existing
> > > Java sources. That way, we could easily configure them as
> > part of the
> > > expansion request (which would contain a JSON
> configuration).
> > I'll
> > > probably fork a thread to discuss this in more detail,
> but would
> > > like to
> > > hear your thoughts.
> > >
> > > -Max
> > >
> > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > Ah, I thought you meant native Flink transforms.
> > > >
> > > > Exactly! The translation code is already there.
> The main
> > > challenge
> > > > is how to
> > > > programmatically configure the BeamIO from
> Python. I
> > suppose
> > > that is
> > > > also an
> > > > unsolved problem for cross-language transforms
> in general.
> > > >
> > > >
> > > > This is what
> https://github.com/apache/beam/pull/7316 does.
> > > >
> > > > For a particular source, one would want to define a
> URN and
> > > > corresponding payload, then (probably) a
> CompositeTransform in
> > > Python
> > > > that takes the users arguments, packages them into the
> > payload,
> > > applies
> > > > the ExternalTransform, and returns the results. How
> to handle
> > > arbitrary
> > > > UDFs embedded in sources is still TBD.
> > > >
> > > > For Matthias' pipeline with PubSubIO we can build
> > something
> > > > specific, but for
> > > > the general case there should be way to
> initialize a
> > Beam IO
> > > via a
> > > > configuration
> > > > map provided by an external environment.
> > > >
> > > >
> > > > I thought quite a bit about how we could represent
> expansions
> > > statically
> > > > (e.g. have some kind of expansion template that
> could be
> > used, at
> > > least
> > > > in many cases, as data without firing up a separate
> > process. May be
> > > > worth doing eventually, but we run into the same issues
> > that were
> > > > discussed at
> > > >
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > >
> > > > If one is already using a portable runner like Flink,
> > having the job
> > > > service process automatically also serve up an
> expansion
> > service for
> > > > various URNs it knows and cares about is probably a
> pretty
> > low bar.
> > > > Flink could serve up things it would rather get back
> > untouched in a
> > > > transform with a special flink runner urn.
> > > >
> > > > As Ahmet mentions, SDF is better solution. I hope
> it's not
> > that far
> > > > away, but even once it comes we'll likely want the
> above
> > > framework to
> > > > invoke the full suite of Java IOs even after they're
> > running on SDF
> > > > themselves.
> > > >
> > > > - Robert
> > > >
> > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > Exactly, that's what I had in mind.
> > > > >
> > > > > A Flink runner native transform would make
> the existing
> > > unbounded
> > > > sources
> > > > > available, similar to:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian
> Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > Wouldn't it be even more useful for the
> transition
> > > period if
> > > > we enabled Beam IO
> > > > > to be used via Flink (like in the legacy
> Flink
> > Runner)? In
> > > > this particular
> > > > > example, Matthias wants to use PubSubIO,
> which
> > is not even
> > > > available as a
> > > > > native
> > > > > Flink transform.
> > > > >
> > > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > > Until SDF is supported, we could also add
> > Flink runner
> > > > native transforms for
> > > > > > selected unbounded sources [1].
> > > > > >
> > > > > > That might be a reasonable option to
> unblock
> > users that
> > > > want to try Python
> > > > > > streaming on Flink.
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 6:51 AM
> Maximilian
> > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > >
> > > > > > > I have a hard time to imagine
> how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers into the existing
> > > > Bounded/UnboundedSource, so I would
> > > > > > love to hear more about the details.
> > > > > >
> > > > > > Isn't it the other way around?
> The SDF is a
> > > > generalization of
> > > > > UnboundedSource.
> > > > > > So we would wrap UnboundedSource
> using
> > SDF. I'm not
> > > > saying it is
> > > > > trivial, but
> > > > > > SDF offers all the functionality that
> > > UnboundedSource
> > > > needs.
> > > > > >
> > > > > > For example, the
> @GetInitialRestriction
> > method
> > > would
> > > > call split on the
> > > > > > UnboundedSource and the restriction
> > trackers would
> > > > then be used to
> > > > > process the
> > > > > > splits.
> > > > > >
> > > > > > On 31.01.19 15:16, Ismaël Mejía
> wrote:
> > > > > > >> Not necessarily. This would
> be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for UnboundedSource.
> Probably the
> > > easier path
> > > > for migration.
> > > > > > >
> > > > > > > That would be fantastic, I
> have heard
> > about such
> > > > wrapper multiple
> > > > > > > times but so far there is not any
> > realistic
> > > > proposal. I have a hard
> > > > > > > time to imagine how can we map
> in a
> > generic way
> > > > RestrictionTrackers
> > > > > > > into the existing
> > Bounded/UnboundedSource, so I
> > > > would love to hear
> > > > > > > more about the details.
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 3:07 PM
> > Maximilian
> > > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >>
> > > > > > >> > In addition to have
> support in
> > the runners,
> > > > this will require a
> > > > > > >> > rewrite of PubsubIO to
> use the
> > new SDF API.
> > > > > > >>
> > > > > > >> Not necessarily. This would
> be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for
> > > > > > >> UnboundedSource. Probably the
> easier
> > path for
> > > > migration.
> > > > > > >>
> > > > > > >> On 31.01.19 14:03, Ismaël
> Mejía wrote:
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java
> IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>
> > > > > > >>> In addition to have support
> in the
> > > runners, this
> > > > will require a
> > > > > > >>> rewrite of PubsubIO to use
> the new
> > SDF API.
> > > > > > >>>
> > > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > > Maximilian Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>
> > > > > > <mailto:mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>>> wrote:
> > > > > > >>>>
> > > > > > >>>> Hi Matthias,
> > > > > > >>>>
> > > > > > >>>> This is already reflected
> in the
> > > compatibility
> > > > matrix, if you look
> > > > > > under SDF.
> > > > > > >>>> There is no UnboundedSource
> > interface for
> > > > portable pipelines.
> > > > > That's a
> > > > > > legacy
> > > > > > >>>> abstraction that will be
> replaced
> > with SDF.
> > > > > > >>>>
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java
> IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> Max
> > > > > > >>>>
> > > > > > >>>> On 31.01.19 12:06, Matthias
> > Baetens wrote:
> > > > > > >>>>> Hey Ankur,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the swift reply.
> > Should I change
> > > > this in the
> > > > > capability matrix
> > > > > > >>>>>
> > > >
> > <https://s.apache.org/apache-beam-portability-support-table>
> > > then?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>> On Thu, 31 Jan 2019 at 09:31,
> > Ankur Goenka
> > > > <goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>
> > > > > > >>>>> <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi Matthias,
> > > > > > >>>>>
> > > > > > >>>>> Unfortunately,
> unbounded reads
> > > including
> > > > pubsub are not yet
> > > > > > supported for
> > > > > > >>>>> portable runners.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>> Ankur
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan 31, 2019
> at 2:44 PM
> > > Matthias
> > > > Baetens
> > > > > > <baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>
> > > > > > >>>>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi everyone,
> > > > > > >>>>>
> > > > > > >>>>> Last few days I
> have been
> > > trying to
> > > > run a streaming
> > > > > > pipeline (code on
> > > > > > >>>>> Github
> > > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > > Flink Runner.
> > > > > > >>>>>
> > > > > > >>>>> I am running a Flink
> > cluster
> > > locally
> > > > (v1.5.6
> > > > > > >>>>>
> > > > <https://flink.apache.org/downloads.html>)
> > > > > > >>>>> I have built the SDK
> > Harness
> > > > Container: /./gradlew
> > > > > > >>>>>
> > > :beam-sdks-python-container:docker/
> > > > > > >>>>> and started the
> JobServer:
> > > /./gradlew
> > > > > > >>>>>
> > > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > > >>>>>
> > -PflinkMasterUrl=localhost:8081./
> > > > > > >>>>>
> > > > > > >>>>> I run my
> pipeline with:
> > > /env/bin/python
> > > > > streaming_pipeline.py
> > > > > > >>>>>
> --runner=PortableRunner
> > > > --job_endpoint=localhost:8099
> > > > > > --output xxx
> > > > > > >>>>>
> --input_subscription xxx
> > > > --output_subscription xxx/
> > > > > > >>>>> /
> > > > > > >>>>> /
> > > > > > >>>>> All this is running
> > inside a
> > > Ubuntu
> > > > (Bionic) in a
> > > > > Virtualbox.
> > > > > > >>>>>
> > > > > > >>>>> The job submits
> fine, but
> > > > unfortunately fails after
> > > > > a few
> > > > > > seconds with
> > > > > > >>>>> the error attached.
> > > > > > >>>>>
> > > > > > >>>>> Anything I am
> missing
> > or doing
> > > wrong?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Chamikara Jayalath <ch...@google.com>.
On Tue, Feb 5, 2019, 8:11 AM Maximilian Michels <mxm@apache.org wrote:
> Good points Cham.
>
> JSON seemed like the most intuitive way to specify a configuration map.
> We already use JSON in other places, e.g. to specify the environment
> configuration. It is not necessarily a contradiction to have JSON inside
> Protobuf. From the perspective of IO authors, the user-friendliness
> plays a role because they wouldn't have to deal with Protobuf.
>
It's a good point that JSON will make this more user-friendly for IO
transforms authors. Probably we should do a bit of experimentation and keep
this experimental in case we hit a performance snag.
>
> I agree that the configuration format is an implementation detail that
> will be hidden to users via easy-to-use wrappers.
>
> Do we have to support UDFs for expanding existing IO? Users would still
> be able to apply UDFs via ParDo on the IO output collections. Generally
> speaking, I can see how for cross-language transforms UDF support would
> be good. For example, a Combine implementation in Java, where the
> combine UDFs come from Python.
>
I think we should try to support UDFs in the first version unless there's a
major blocker that hinders realizing this. Many IO transforms available
today expect users to pass UDFs to realize various features (for example,
dynamic destinations for BigqueryIO and FileIO, timestamp function for
KafkaIO). I think without support for UDFs usability of cross-language
transforms feature will be significantly limited.
I'm happy to work with you to realize this.
Thanks,
Cham
> I suppose the question is, do we try to solve the general case, or do we
> go with a simpler approach for enabling the use of existing IO first?
> Lack of IO seems to be the most pressing issue for the adoption of Beam
> Python. I imagine that a backwards-compatible incremental support for
> cross-language transforms (IOs first, later other transforms) would be
> possible.
>
> -Max
>
> On 05.02.19 03:07, Chamikara Jayalath wrote:
> >
> >
> > On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Yes, I imagine sources to implement a JsonConfigurable interface
> (e.g.
> > on their builders):
> >
> > JsonConfigurable {
> > // Either a json string or Map<String, String>
> > apply(String jsonConfig);
> > }
> >
> > In Python we would create this transform:
> >
> > URN: JsonConfiguredSource:v1
> > payload: {
> > environment: environment_id, // Java/Python/Go
> > resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
> > configuration: json config, // { "topic" : "my_pubsub_topic" }
> > }
> >
> >
> > Thanks Max, this is a great first step towards defining to API for
> > cross-language transforms.
> > Is there a reason why you would want to use JSON instead of a proto
> > here. I guess we'll be providing a more user friendly language wrapper
> > (for example, Python) for end-users here, so user-friendliness-wise, the
> > format we choose won't matter much (for pipeline authors).
> > If we don't support UDFs, performance difference will be negligible, but
> > UDFs might require a callback to original SDK (per-element worst case).
> > So might make sense to choose the more efficient format.
> >
> > Also, probably we need to define a more expanded definition (proto/JSON)
> > to support UDFs. For example, a payload + a set of parameter definitions
> > so that the target SDK (for example, Java) can call back the original
> > SDK where the pipeline was authored in (for example, Python) to resolve
> > UDFs at runtime.
> >
> > Thanks,
> > Cham
> >
> > That's more generic and could be used for other languages where we
> > might
> > have sources/sinks.
> >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi protos that fully specified it, and we use that same
> > format to translate back out.)
> >
> > Not that I know of.
> >
> > On 01.02.19 14:02, Robert Bradshaw wrote:
> > > Are you suggesting something akin to a generic
> > >
> > > urn: JsonConfiguredJavaSource
> > > payload: some json specifying which source and which
> parameters
> > >
> > > which would expand to actually constructing and applying that
> source?
> > >
> > > (FWIW, I was imagining PubSubIO already had a translation into
> > BeamFnApi
> > > protos that fully specified it, and we use that same format to
> > translate
> > > back out.)
> > >
> > > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Recaping here:
> > >
> > > We all agree that SDF is the way to go for future
> > implementations of
> > > sources. It enables us to get rid of the source interfaces.
> > However,
> > > SDF
> > > does not solve the lack of streaming sources in Python.
> > >
> > > The expansion PR (thanks btw!) solves the problem of
> > > expanding/translating URNs known to an ExpansionService. That
> > is a more
> > > programmatic way of replacing language-specific performs,
> > instead of
> > > relying on translators directly in the Runner.
> > >
> > > What is unsolved is the configuration of sources from a
> foreign
> > > environment. In my opinion this is the most pressing issue
> > for Python
> > > sources, because what is PubSubIO worth in Python if you
> cannot
> > > configure it?
> > >
> > > What about this:
> > >
> > > I think it is worth adding a JSON configuration option for
> > all existing
> > > Java sources. That way, we could easily configure them as
> > part of the
> > > expansion request (which would contain a JSON configuration).
> > I'll
> > > probably fork a thread to discuss this in more detail, but
> would
> > > like to
> > > hear your thoughts.
> > >
> > > -Max
> > >
> > > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Ah, I thought you meant native Flink transforms.
> > > >
> > > > Exactly! The translation code is already there. The
> main
> > > challenge
> > > > is how to
> > > > programmatically configure the BeamIO from Python. I
> > suppose
> > > that is
> > > > also an
> > > > unsolved problem for cross-language transforms in
> general.
> > > >
> > > >
> > > > This is what https://github.com/apache/beam/pull/7316
> does.
> > > >
> > > > For a particular source, one would want to define a URN and
> > > > corresponding payload, then (probably) a
> CompositeTransform in
> > > Python
> > > > that takes the users arguments, packages them into the
> > payload,
> > > applies
> > > > the ExternalTransform, and returns the results. How to
> handle
> > > arbitrary
> > > > UDFs embedded in sources is still TBD.
> > > >
> > > > For Matthias' pipeline with PubSubIO we can build
> > something
> > > > specific, but for
> > > > the general case there should be way to initialize a
> > Beam IO
> > > via a
> > > > configuration
> > > > map provided by an external environment.
> > > >
> > > >
> > > > I thought quite a bit about how we could represent
> expansions
> > > statically
> > > > (e.g. have some kind of expansion template that could be
> > used, at
> > > least
> > > > in many cases, as data without firing up a separate
> > process. May be
> > > > worth doing eventually, but we run into the same issues
> > that were
> > > > discussed at
> > > >
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > > >
> > > > If one is already using a portable runner like Flink,
> > having the job
> > > > service process automatically also serve up an expansion
> > service for
> > > > various URNs it knows and cares about is probably a pretty
> > low bar.
> > > > Flink could serve up things it would rather get back
> > untouched in a
> > > > transform with a special flink runner urn.
> > > >
> > > > As Ahmet mentions, SDF is better solution. I hope it's not
> > that far
> > > > away, but even once it comes we'll likely want the above
> > > framework to
> > > > invoke the full suite of Java IOs even after they're
> > running on SDF
> > > > themselves.
> > > >
> > > > - Robert
> > > >
> > > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > > Exactly, that's what I had in mind.
> > > > >
> > > > > A Flink runner native transform would make the
> existing
> > > unbounded
> > > > sources
> > > > > available, similar to:
> > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > Wouldn't it be even more useful for the
> transition
> > > period if
> > > > we enabled Beam IO
> > > > > to be used via Flink (like in the legacy Flink
> > Runner)? In
> > > > this particular
> > > > > example, Matthias wants to use PubSubIO, which
> > is not even
> > > > available as a
> > > > > native
> > > > > Flink transform.
> > > > >
> > > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > > Until SDF is supported, we could also add
> > Flink runner
> > > > native transforms for
> > > > > > selected unbounded sources [1].
> > > > > >
> > > > > > That might be a reasonable option to unblock
> > users that
> > > > want to try Python
> > > > > > streaming on Flink.
> > > > > >
> > > > > > Thomas
> > > > > >
> > > > > > [1]
> > > > > >
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > > >
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian
> > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >
> > > > > > > I have a hard time to imagine how can
> > we map
> > > in a
> > > > generic way
> > > > > > RestrictionTrackers into the existing
> > > > Bounded/UnboundedSource, so I would
> > > > > > love to hear more about the details.
> > > > > >
> > > > > > Isn't it the other way around? The SDF
> is a
> > > > generalization of
> > > > > UnboundedSource.
> > > > > > So we would wrap UnboundedSource using
> > SDF. I'm not
> > > > saying it is
> > > > > trivial, but
> > > > > > SDF offers all the functionality that
> > > UnboundedSource
> > > > needs.
> > > > > >
> > > > > > For example, the @GetInitialRestriction
> > method
> > > would
> > > > call split on the
> > > > > > UnboundedSource and the restriction
> > trackers would
> > > > then be used to
> > > > > process the
> > > > > > splits.
> > > > > >
> > > > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > > > >> Not necessarily. This would be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for UnboundedSource. Probably the
> > > easier path
> > > > for migration.
> > > > > > >
> > > > > > > That would be fantastic, I have heard
> > about such
> > > > wrapper multiple
> > > > > > > times but so far there is not any
> > realistic
> > > > proposal. I have a hard
> > > > > > > time to imagine how can we map in a
> > generic way
> > > > RestrictionTrackers
> > > > > > > into the existing
> > Bounded/UnboundedSource, so I
> > > > would love to hear
> > > > > > > more about the details.
> > > > > > >
> > > > > > > On Thu, Jan 31, 2019 at 3:07 PM
> > Maximilian
> > > Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > > >>
> > > > > > >> > In addition to have support in
> > the runners,
> > > > this will require a
> > > > > > >> > rewrite of PubsubIO to use the
> > new SDF API.
> > > > > > >>
> > > > > > >> Not necessarily. This would be one
> > way. Another
> > > > way is build an SDF
> > > > > > wrapper for
> > > > > > >> UnboundedSource. Probably the easier
> > path for
> > > > migration.
> > > > > > >>
> > > > > > >> On 31.01.19 14:03, Ismaël Mejía
> wrote:
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>
> > > > > > >>> In addition to have support in the
> > > runners, this
> > > > will require a
> > > > > > >>> rewrite of PubsubIO to use the new
> > SDF API.
> > > > > > >>>
> > > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > > Maximilian Michels
> > > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > > <mailto:mxm@apache.org
> > <ma...@apache.org> <mailto:mxm@apache.org <mailto:
> mxm@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > > >>>>
> > > > > > >>>> Hi Matthias,
> > > > > > >>>>
> > > > > > >>>> This is already reflected in the
> > > compatibility
> > > > matrix, if you look
> > > > > > under SDF.
> > > > > > >>>> There is no UnboundedSource
> > interface for
> > > > portable pipelines.
> > > > > That's a
> > > > > > legacy
> > > > > > >>>> abstraction that will be replaced
> > with SDF.
> > > > > > >>>>
> > > > > > >>>> Fortunately, there is already a
> > pending
> > > PR for
> > > > cross-language
> > > > > > pipelines which
> > > > > > >>>> will allow us to use Java IO like
> > PubSub in
> > > > Python jobs.
> > > > > > >>>>
> > > > > > >>>> Thanks,
> > > > > > >>>> Max
> > > > > > >>>>
> > > > > > >>>> On 31.01.19 12:06, Matthias
> > Baetens wrote:
> > > > > > >>>>> Hey Ankur,
> > > > > > >>>>>
> > > > > > >>>>> Thanks for the swift reply.
> > Should I change
> > > > this in the
> > > > > capability matrix
> > > > > > >>>>>
> > > >
> > <https://s.apache.org/apache-beam-portability-support-table>
> > > then?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>> On Thu, 31 Jan 2019 at 09:31,
> > Ankur Goenka
> > > > <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > > <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi Matthias,
> > > > > > >>>>>
> > > > > > >>>>> Unfortunately, unbounded
> reads
> > > including
> > > > pubsub are not yet
> > > > > > supported for
> > > > > > >>>>> portable runners.
> > > > > > >>>>>
> > > > > > >>>>> Thanks,
> > > > > > >>>>> Ankur
> > > > > > >>>>>
> > > > > > >>>>> On Thu, Jan 31, 2019 at
> 2:44 PM
> > > Matthias
> > > > Baetens
> > > > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>
> > > > > > >>>>>
> > <mailto:baetensmatthias@gmail.com <mailto:
> baetensmatthias@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>>> wrote:
> > > > > > >>>>>
> > > > > > >>>>> Hi everyone,
> > > > > > >>>>>
> > > > > > >>>>> Last few days I have
> been
> > > trying to
> > > > run a streaming
> > > > > > pipeline (code on
> > > > > > >>>>> Github
> > > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > > Flink Runner.
> > > > > > >>>>>
> > > > > > >>>>> I am running a Flink
> > cluster
> > > locally
> > > > (v1.5.6
> > > > > > >>>>>
> > > > <https://flink.apache.org/downloads.html>)
> > > > > > >>>>> I have built the SDK
> > Harness
> > > > Container: /./gradlew
> > > > > > >>>>>
> > > :beam-sdks-python-container:docker/
> > > > > > >>>>> and started the
> JobServer:
> > > /./gradlew
> > > > > > >>>>>
> > > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > > >>>>>
> > -PflinkMasterUrl=localhost:8081./
> > > > > > >>>>>
> > > > > > >>>>> I run my pipeline with:
> > > /env/bin/python
> > > > > streaming_pipeline.py
> > > > > > >>>>> --runner=PortableRunner
> > > > --job_endpoint=localhost:8099
> > > > > > --output xxx
> > > > > > >>>>> --input_subscription xxx
> > > > --output_subscription xxx/
> > > > > > >>>>> /
> > > > > > >>>>> /
> > > > > > >>>>> All this is running
> > inside a
> > > Ubuntu
> > > > (Bionic) in a
> > > > > Virtualbox.
> > > > > > >>>>>
> > > > > > >>>>> The job submits fine,
> but
> > > > unfortunately fails after
> > > > > a few
> > > > > > seconds with
> > > > > > >>>>> the error attached.
> > > > > > >>>>>
> > > > > > >>>>> Anything I am missing
> > or doing
> > > wrong?
> > > > > > >>>>>
> > > > > > >>>>> Many thanks.
> > > > > > >>>>> Best,
> > > > > > >>>>> Matthias
> > > > > > >>>>>
> > > > > > >>>>>
> > > > > >
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Good points Cham.
JSON seemed like the most intuitive way to specify a configuration map.
We already use JSON in other places, e.g. to specify the environment
configuration. It is not necessarily a contradiction to have JSON inside
Protobuf. From the perspective of IO authors, the user-friendliness
plays a role because they wouldn't have to deal with Protobuf.
I agree that the configuration format is an implementation detail that
will be hidden to users via easy-to-use wrappers.
Do we have to support UDFs for expanding existing IO? Users would still
be able to apply UDFs via ParDo on the IO output collections. Generally
speaking, I can see how for cross-language transforms UDF support would
be good. For example, a Combine implementation in Java, where the
combine UDFs come from Python.
I suppose the question is, do we try to solve the general case, or do we
go with a simpler approach for enabling the use of existing IO first?
Lack of IO seems to be the most pressing issue for the adoption of Beam
Python. I imagine that a backwards-compatible incremental support for
cross-language transforms (IOs first, later other transforms) would be
possible.
-Max
On 05.02.19 03:07, Chamikara Jayalath wrote:
>
>
> On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Yes, I imagine sources to implement a JsonConfigurable interface (e.g.
> on their builders):
>
> JsonConfigurable {
> // Either a json string or Map<String, String>
> apply(String jsonConfig);
> }
>
> In Python we would create this transform:
>
> URN: JsonConfiguredSource:v1
> payload: {
> environment: environment_id, // Java/Python/Go
> resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
> configuration: json config, // { "topic" : "my_pubsub_topic" }
> }
>
>
> Thanks Max, this is a great first step towards defining to API for
> cross-language transforms.
> Is there a reason why you would want to use JSON instead of a proto
> here. I guess we'll be providing a more user friendly language wrapper
> (for example, Python) for end-users here, so user-friendliness-wise, the
> format we choose won't matter much (for pipeline authors).
> If we don't support UDFs, performance difference will be negligible, but
> UDFs might require a callback to original SDK (per-element worst case).
> So might make sense to choose the more efficient format.
>
> Also, probably we need to define a more expanded definition (proto/JSON)
> to support UDFs. For example, a payload + a set of parameter definitions
> so that the target SDK (for example, Java) can call back the original
> SDK where the pipeline was authored in (for example, Python) to resolve
> UDFs at runtime.
>
> Thanks,
> Cham
>
> That's more generic and could be used for other languages where we
> might
> have sources/sinks.
>
> > (FWIW, I was imagining PubSubIO already had a translation into
> BeamFnApi protos that fully specified it, and we use that same
> format to translate back out.)
>
> Not that I know of.
>
> On 01.02.19 14:02, Robert Bradshaw wrote:
> > Are you suggesting something akin to a generic
> >
> > urn: JsonConfiguredJavaSource
> > payload: some json specifying which source and which parameters
> >
> > which would expand to actually constructing and applying that source?
> >
> > (FWIW, I was imagining PubSubIO already had a translation into
> BeamFnApi
> > protos that fully specified it, and we use that same format to
> translate
> > back out.)
> >
> > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Recaping here:
> >
> > We all agree that SDF is the way to go for future
> implementations of
> > sources. It enables us to get rid of the source interfaces.
> However,
> > SDF
> > does not solve the lack of streaming sources in Python.
> >
> > The expansion PR (thanks btw!) solves the problem of
> > expanding/translating URNs known to an ExpansionService. That
> is a more
> > programmatic way of replacing language-specific performs,
> instead of
> > relying on translators directly in the Runner.
> >
> > What is unsolved is the configuration of sources from a foreign
> > environment. In my opinion this is the most pressing issue
> for Python
> > sources, because what is PubSubIO worth in Python if you cannot
> > configure it?
> >
> > What about this:
> >
> > I think it is worth adding a JSON configuration option for
> all existing
> > Java sources. That way, we could easily configure them as
> part of the
> > expansion request (which would contain a JSON configuration).
> I'll
> > probably fork a thread to discuss this in more detail, but would
> > like to
> > hear your thoughts.
> >
> > -Max
> >
> > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Ah, I thought you meant native Flink transforms.
> > >
> > > Exactly! The translation code is already there. The main
> > challenge
> > > is how to
> > > programmatically configure the BeamIO from Python. I
> suppose
> > that is
> > > also an
> > > unsolved problem for cross-language transforms in general.
> > >
> > >
> > > This is what https://github.com/apache/beam/pull/7316 does.
> > >
> > > For a particular source, one would want to define a URN and
> > > corresponding payload, then (probably) a CompositeTransform in
> > Python
> > > that takes the users arguments, packages them into the
> payload,
> > applies
> > > the ExternalTransform, and returns the results. How to handle
> > arbitrary
> > > UDFs embedded in sources is still TBD.
> > >
> > > For Matthias' pipeline with PubSubIO we can build
> something
> > > specific, but for
> > > the general case there should be way to initialize a
> Beam IO
> > via a
> > > configuration
> > > map provided by an external environment.
> > >
> > >
> > > I thought quite a bit about how we could represent expansions
> > statically
> > > (e.g. have some kind of expansion template that could be
> used, at
> > least
> > > in many cases, as data without firing up a separate
> process. May be
> > > worth doing eventually, but we run into the same issues
> that were
> > > discussed at
> > >
> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > >
> > > If one is already using a portable runner like Flink,
> having the job
> > > service process automatically also serve up an expansion
> service for
> > > various URNs it knows and cares about is probably a pretty
> low bar.
> > > Flink could serve up things it would rather get back
> untouched in a
> > > transform with a special flink runner urn.
> > >
> > > As Ahmet mentions, SDF is better solution. I hope it's not
> that far
> > > away, but even once it comes we'll likely want the above
> > framework to
> > > invoke the full suite of Java IOs even after they're
> running on SDF
> > > themselves.
> > >
> > > - Robert
> > >
> > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > Exactly, that's what I had in mind.
> > > >
> > > > A Flink runner native transform would make the existing
> > unbounded
> > > sources
> > > > available, similar to:
> > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > Wouldn't it be even more useful for the transition
> > period if
> > > we enabled Beam IO
> > > > to be used via Flink (like in the legacy Flink
> Runner)? In
> > > this particular
> > > > example, Matthias wants to use PubSubIO, which
> is not even
> > > available as a
> > > > native
> > > > Flink transform.
> > > >
> > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > Until SDF is supported, we could also add
> Flink runner
> > > native transforms for
> > > > > selected unbounded sources [1].
> > > > >
> > > > > That might be a reasonable option to unblock
> users that
> > > want to try Python
> > > > > streaming on Flink.
> > > > >
> > > > > Thomas
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian
> Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > >
> > > > > > I have a hard time to imagine how can
> we map
> > in a
> > > generic way
> > > > > RestrictionTrackers into the existing
> > > Bounded/UnboundedSource, so I would
> > > > > love to hear more about the details.
> > > > >
> > > > > Isn't it the other way around? The SDF is a
> > > generalization of
> > > > UnboundedSource.
> > > > > So we would wrap UnboundedSource using
> SDF. I'm not
> > > saying it is
> > > > trivial, but
> > > > > SDF offers all the functionality that
> > UnboundedSource
> > > needs.
> > > > >
> > > > > For example, the @GetInitialRestriction
> method
> > would
> > > call split on the
> > > > > UnboundedSource and the restriction
> trackers would
> > > then be used to
> > > > process the
> > > > > splits.
> > > > >
> > > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > > >> Not necessarily. This would be one
> way. Another
> > > way is build an SDF
> > > > > wrapper for UnboundedSource. Probably the
> > easier path
> > > for migration.
> > > > > >
> > > > > > That would be fantastic, I have heard
> about such
> > > wrapper multiple
> > > > > > times but so far there is not any
> realistic
> > > proposal. I have a hard
> > > > > > time to imagine how can we map in a
> generic way
> > > RestrictionTrackers
> > > > > > into the existing
> Bounded/UnboundedSource, so I
> > > would love to hear
> > > > > > more about the details.
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 3:07 PM
> Maximilian
> > Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >>
> > > > > >> > In addition to have support in
> the runners,
> > > this will require a
> > > > > >> > rewrite of PubsubIO to use the
> new SDF API.
> > > > > >>
> > > > > >> Not necessarily. This would be one
> way. Another
> > > way is build an SDF
> > > > > wrapper for
> > > > > >> UnboundedSource. Probably the easier
> path for
> > > migration.
> > > > > >>
> > > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > > >>>> Fortunately, there is already a
> pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like
> PubSub in
> > > Python jobs.
> > > > > >>>
> > > > > >>> In addition to have support in the
> > runners, this
> > > will require a
> > > > > >>> rewrite of PubsubIO to use the new
> SDF API.
> > > > > >>>
> > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>
> > > > > <mailto:mxm@apache.org
> <ma...@apache.org> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>>> wrote:
> > > > > >>>>
> > > > > >>>> Hi Matthias,
> > > > > >>>>
> > > > > >>>> This is already reflected in the
> > compatibility
> > > matrix, if you look
> > > > > under SDF.
> > > > > >>>> There is no UnboundedSource
> interface for
> > > portable pipelines.
> > > > That's a
> > > > > legacy
> > > > > >>>> abstraction that will be replaced
> with SDF.
> > > > > >>>>
> > > > > >>>> Fortunately, there is already a
> pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like
> PubSub in
> > > Python jobs.
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> Max
> > > > > >>>>
> > > > > >>>> On 31.01.19 12:06, Matthias
> Baetens wrote:
> > > > > >>>>> Hey Ankur,
> > > > > >>>>>
> > > > > >>>>> Thanks for the swift reply.
> Should I change
> > > this in the
> > > > capability matrix
> > > > > >>>>>
> > >
> <https://s.apache.org/apache-beam-portability-support-table>
> > then?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>> On Thu, 31 Jan 2019 at 09:31,
> Ankur Goenka
> > > <goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>
> > > > > >>>>> <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi Matthias,
> > > > > >>>>>
> > > > > >>>>> Unfortunately, unbounded reads
> > including
> > > pubsub are not yet
> > > > > supported for
> > > > > >>>>> portable runners.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>> Ankur
> > > > > >>>>>
> > > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> > Matthias
> > > Baetens
> > > > > <baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>
> > > > > >>>>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi everyone,
> > > > > >>>>>
> > > > > >>>>> Last few days I have been
> > trying to
> > > run a streaming
> > > > > pipeline (code on
> > > > > >>>>> Github
> > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > Flink Runner.
> > > > > >>>>>
> > > > > >>>>> I am running a Flink
> cluster
> > locally
> > > (v1.5.6
> > > > > >>>>>
> > > <https://flink.apache.org/downloads.html>)
> > > > > >>>>> I have built the SDK
> Harness
> > > Container: /./gradlew
> > > > > >>>>>
> > :beam-sdks-python-container:docker/
> > > > > >>>>> and started the JobServer:
> > /./gradlew
> > > > > >>>>>
> > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > >>>>>
> -PflinkMasterUrl=localhost:8081./
> > > > > >>>>>
> > > > > >>>>> I run my pipeline with:
> > /env/bin/python
> > > > streaming_pipeline.py
> > > > > >>>>> --runner=PortableRunner
> > > --job_endpoint=localhost:8099
> > > > > --output xxx
> > > > > >>>>> --input_subscription xxx
> > > --output_subscription xxx/
> > > > > >>>>> /
> > > > > >>>>> /
> > > > > >>>>> All this is running
> inside a
> > Ubuntu
> > > (Bionic) in a
> > > > Virtualbox.
> > > > > >>>>>
> > > > > >>>>> The job submits fine, but
> > > unfortunately fails after
> > > > a few
> > > > > seconds with
> > > > > >>>>> the error attached.
> > > > > >>>>>
> > > > > >>>>> Anything I am missing
> or doing
> > wrong?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>>
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Chamikara Jayalath <ch...@google.com>.
On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mx...@apache.org> wrote:
> Yes, I imagine sources to implement a JsonConfigurable interface (e.g.
> on their builders):
>
> JsonConfigurable {
> // Either a json string or Map<String, String>
> apply(String jsonConfig);
> }
>
> In Python we would create this transform:
>
> URN: JsonConfiguredSource:v1
> payload: {
> environment: environment_id, // Java/Python/Go
> resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
> configuration: json config, // { "topic" : "my_pubsub_topic" }
> }
>
>
Thanks Max, this is a great first step towards defining to API for
cross-language transforms.
Is there a reason why you would want to use JSON instead of a proto here. I
guess we'll be providing a more user friendly language wrapper (for
example, Python) for end-users here, so user-friendliness-wise, the format
we choose won't matter much (for pipeline authors).
If we don't support UDFs, performance difference will be negligible, but
UDFs might require a callback to original SDK (per-element worst case). So
might make sense to choose the more efficient format.
Also, probably we need to define a more expanded definition (proto/JSON) to
support UDFs. For example, a payload + a set of parameter definitions so
that the target SDK (for example, Java) can call back the original SDK
where the pipeline was authored in (for example, Python) to resolve UDFs at
runtime.
Thanks,
Cham
> That's more generic and could be used for other languages where we might
> have sources/sinks.
>
> > (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
> protos that fully specified it, and we use that same format to translate
> back out.)
>
> Not that I know of.
>
> On 01.02.19 14:02, Robert Bradshaw wrote:
> > Are you suggesting something akin to a generic
> >
> > urn: JsonConfiguredJavaSource
> > payload: some json specifying which source and which parameters
> >
> > which would expand to actually constructing and applying that source?
> >
> > (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
> > protos that fully specified it, and we use that same format to translate
> > back out.)
> >
> > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Recaping here:
> >
> > We all agree that SDF is the way to go for future implementations of
> > sources. It enables us to get rid of the source interfaces. However,
> > SDF
> > does not solve the lack of streaming sources in Python.
> >
> > The expansion PR (thanks btw!) solves the problem of
> > expanding/translating URNs known to an ExpansionService. That is a
> more
> > programmatic way of replacing language-specific performs, instead of
> > relying on translators directly in the Runner.
> >
> > What is unsolved is the configuration of sources from a foreign
> > environment. In my opinion this is the most pressing issue for Python
> > sources, because what is PubSubIO worth in Python if you cannot
> > configure it?
> >
> > What about this:
> >
> > I think it is worth adding a JSON configuration option for all
> existing
> > Java sources. That way, we could easily configure them as part of the
> > expansion request (which would contain a JSON configuration). I'll
> > probably fork a thread to discuss this in more detail, but would
> > like to
> > hear your thoughts.
> >
> > -Max
> >
> > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Ah, I thought you meant native Flink transforms.
> > >
> > > Exactly! The translation code is already there. The main
> > challenge
> > > is how to
> > > programmatically configure the BeamIO from Python. I suppose
> > that is
> > > also an
> > > unsolved problem for cross-language transforms in general.
> > >
> > >
> > > This is what https://github.com/apache/beam/pull/7316 does.
> > >
> > > For a particular source, one would want to define a URN and
> > > corresponding payload, then (probably) a CompositeTransform in
> > Python
> > > that takes the users arguments, packages them into the payload,
> > applies
> > > the ExternalTransform, and returns the results. How to handle
> > arbitrary
> > > UDFs embedded in sources is still TBD.
> > >
> > > For Matthias' pipeline with PubSubIO we can build something
> > > specific, but for
> > > the general case there should be way to initialize a Beam IO
> > via a
> > > configuration
> > > map provided by an external environment.
> > >
> > >
> > > I thought quite a bit about how we could represent expansions
> > statically
> > > (e.g. have some kind of expansion template that could be used, at
> > least
> > > in many cases, as data without firing up a separate process. May
> be
> > > worth doing eventually, but we run into the same issues that were
> > > discussed at
> > > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > >
> > > If one is already using a portable runner like Flink, having the
> job
> > > service process automatically also serve up an expansion service
> for
> > > various URNs it knows and cares about is probably a pretty low
> bar.
> > > Flink could serve up things it would rather get back untouched in
> a
> > > transform with a special flink runner urn.
> > >
> > > As Ahmet mentions, SDF is better solution. I hope it's not that
> far
> > > away, but even once it comes we'll likely want the above
> > framework to
> > > invoke the full suite of Java IOs even after they're running on
> SDF
> > > themselves.
> > >
> > > - Robert
> > >
> > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > Exactly, that's what I had in mind.
> > > >
> > > > A Flink runner native transform would make the existing
> > unbounded
> > > sources
> > > > available, similar to:
> > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Wouldn't it be even more useful for the transition
> > period if
> > > we enabled Beam IO
> > > > to be used via Flink (like in the legacy Flink
> Runner)? In
> > > this particular
> > > > example, Matthias wants to use PubSubIO, which is not
> even
> > > available as a
> > > > native
> > > > Flink transform.
> > > >
> > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > Until SDF is supported, we could also add Flink
> runner
> > > native transforms for
> > > > > selected unbounded sources [1].
> > > > >
> > > > > That might be a reasonable option to unblock users
> that
> > > want to try Python
> > > > > streaming on Flink.
> > > > >
> > > > > Thomas
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > > I have a hard time to imagine how can we map
> > in a
> > > generic way
> > > > > RestrictionTrackers into the existing
> > > Bounded/UnboundedSource, so I would
> > > > > love to hear more about the details.
> > > > >
> > > > > Isn't it the other way around? The SDF is a
> > > generalization of
> > > > UnboundedSource.
> > > > > So we would wrap UnboundedSource using SDF. I'm
> not
> > > saying it is
> > > > trivial, but
> > > > > SDF offers all the functionality that
> > UnboundedSource
> > > needs.
> > > > >
> > > > > For example, the @GetInitialRestriction method
> > would
> > > call split on the
> > > > > UnboundedSource and the restriction trackers
> would
> > > then be used to
> > > > process the
> > > > > splits.
> > > > >
> > > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > > >> Not necessarily. This would be one way.
> Another
> > > way is build an SDF
> > > > > wrapper for UnboundedSource. Probably the
> > easier path
> > > for migration.
> > > > > >
> > > > > > That would be fantastic, I have heard about
> such
> > > wrapper multiple
> > > > > > times but so far there is not any realistic
> > > proposal. I have a hard
> > > > > > time to imagine how can we map in a generic
> way
> > > RestrictionTrackers
> > > > > > into the existing Bounded/UnboundedSource,
> so I
> > > would love to hear
> > > > > > more about the details.
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian
> > Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > > >>
> > > > > >> > In addition to have support in the
> runners,
> > > this will require a
> > > > > >> > rewrite of PubsubIO to use the new SDF
> API.
> > > > > >>
> > > > > >> Not necessarily. This would be one way.
> Another
> > > way is build an SDF
> > > > > wrapper for
> > > > > >> UnboundedSource. Probably the easier path
> for
> > > migration.
> > > > > >>
> > > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > > >>>> Fortunately, there is already a pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like PubSub
> in
> > > Python jobs.
> > > > > >>>
> > > > > >>> In addition to have support in the
> > runners, this
> > > will require a
> > > > > >>> rewrite of PubsubIO to use the new SDF API.
> > > > > >>>
> > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > > >>>>
> > > > > >>>> Hi Matthias,
> > > > > >>>>
> > > > > >>>> This is already reflected in the
> > compatibility
> > > matrix, if you look
> > > > > under SDF.
> > > > > >>>> There is no UnboundedSource interface for
> > > portable pipelines.
> > > > That's a
> > > > > legacy
> > > > > >>>> abstraction that will be replaced with
> SDF.
> > > > > >>>>
> > > > > >>>> Fortunately, there is already a pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like PubSub
> in
> > > Python jobs.
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> Max
> > > > > >>>>
> > > > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > > > >>>>> Hey Ankur,
> > > > > >>>>>
> > > > > >>>>> Thanks for the swift reply. Should I
> change
> > > this in the
> > > > capability matrix
> > > > > >>>>>
> > > <https://s.apache.org/apache-beam-portability-support-table>
> > then?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur
> Goenka
> > > <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi Matthias,
> > > > > >>>>>
> > > > > >>>>> Unfortunately, unbounded reads
> > including
> > > pubsub are not yet
> > > > > supported for
> > > > > >>>>> portable runners.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>> Ankur
> > > > > >>>>>
> > > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> > Matthias
> > > Baetens
> > > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi everyone,
> > > > > >>>>>
> > > > > >>>>> Last few days I have been
> > trying to
> > > run a streaming
> > > > > pipeline (code on
> > > > > >>>>> Github
> > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > Flink Runner.
> > > > > >>>>>
> > > > > >>>>> I am running a Flink cluster
> > locally
> > > (v1.5.6
> > > > > >>>>>
> > > <https://flink.apache.org/downloads.html>)
> > > > > >>>>> I have built the SDK Harness
> > > Container: /./gradlew
> > > > > >>>>>
> > :beam-sdks-python-container:docker/
> > > > > >>>>> and started the JobServer:
> > /./gradlew
> > > > > >>>>>
> > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > >>>>>
> -PflinkMasterUrl=localhost:8081./
> > > > > >>>>>
> > > > > >>>>> I run my pipeline with:
> > /env/bin/python
> > > > streaming_pipeline.py
> > > > > >>>>> --runner=PortableRunner
> > > --job_endpoint=localhost:8099
> > > > > --output xxx
> > > > > >>>>> --input_subscription xxx
> > > --output_subscription xxx/
> > > > > >>>>> /
> > > > > >>>>> /
> > > > > >>>>> All this is running inside a
> > Ubuntu
> > > (Bionic) in a
> > > > Virtualbox.
> > > > > >>>>>
> > > > > >>>>> The job submits fine, but
> > > unfortunately fails after
> > > > a few
> > > > > seconds with
> > > > > >>>>> the error attached.
> > > > > >>>>>
> > > > > >>>>> Anything I am missing or doing
> > wrong?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>>
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Thomas Weise <th...@gmail.com>.
On Fri, Feb 1, 2019 at 6:12 AM Maximilian Michels <mx...@apache.org> wrote:
> Yes, I imagine sources to implement a JsonConfigurable interface (e.g.
> on their builders):
>
> JsonConfigurable {
> // Either a json string or Map<String, String>
> apply(String jsonConfig);
> }
>
> In Python we would create this transform:
>
> URN: JsonConfiguredSource:v1
> payload: {
> environment: environment_id, // Java/Python/Go
> resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
> configuration: json config, // { "topic" : "my_pubsub_topic" }
> }
>
> That's more generic and could be used for other languages where we might
> have sources/sinks.
>
Looks good!
I imagine there would be a wrapper for this similar to:
https://github.com/apache/beam/blob/master/sdks/python/apache_beam/io/flink/flink_streaming_impulse_source.py#L33
> > (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
> protos that fully specified it, and we use that same format to translate
> back out.)
>
> Not that I know of.
>
> On 01.02.19 14:02, Robert Bradshaw wrote:
> > Are you suggesting something akin to a generic
> >
> > urn: JsonConfiguredJavaSource
> > payload: some json specifying which source and which parameters
> >
> > which would expand to actually constructing and applying that source?
> >
> > (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
> > protos that fully specified it, and we use that same format to translate
> > back out.)
> >
> > On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Recaping here:
> >
> > We all agree that SDF is the way to go for future implementations of
> > sources. It enables us to get rid of the source interfaces. However,
> > SDF
> > does not solve the lack of streaming sources in Python.
> >
> > The expansion PR (thanks btw!) solves the problem of
> > expanding/translating URNs known to an ExpansionService. That is a
> more
> > programmatic way of replacing language-specific performs, instead of
> > relying on translators directly in the Runner.
> >
> > What is unsolved is the configuration of sources from a foreign
> > environment. In my opinion this is the most pressing issue for Python
> > sources, because what is PubSubIO worth in Python if you cannot
> > configure it?
> >
> > What about this:
> >
> > I think it is worth adding a JSON configuration option for all
> existing
> > Java sources. That way, we could easily configure them as part of the
> > expansion request (which would contain a JSON configuration). I'll
> > probably fork a thread to discuss this in more detail, but would
> > like to
> > hear your thoughts.
> >
> > -Max
> >
> > On 01.02.19 13:08, Robert Bradshaw wrote:
> > > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Ah, I thought you meant native Flink transforms.
> > >
> > > Exactly! The translation code is already there. The main
> > challenge
> > > is how to
> > > programmatically configure the BeamIO from Python. I suppose
> > that is
> > > also an
> > > unsolved problem for cross-language transforms in general.
> > >
> > >
> > > This is what https://github.com/apache/beam/pull/7316 does.
> > >
> > > For a particular source, one would want to define a URN and
> > > corresponding payload, then (probably) a CompositeTransform in
> > Python
> > > that takes the users arguments, packages them into the payload,
> > applies
> > > the ExternalTransform, and returns the results. How to handle
> > arbitrary
> > > UDFs embedded in sources is still TBD.
> > >
> > > For Matthias' pipeline with PubSubIO we can build something
> > > specific, but for
> > > the general case there should be way to initialize a Beam IO
> > via a
> > > configuration
> > > map provided by an external environment.
> > >
> > >
> > > I thought quite a bit about how we could represent expansions
> > statically
> > > (e.g. have some kind of expansion template that could be used, at
> > least
> > > in many cases, as data without firing up a separate process. May
> be
> > > worth doing eventually, but we run into the same issues that were
> > > discussed at
> > > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> > >
> > > If one is already using a portable runner like Flink, having the
> job
> > > service process automatically also serve up an expansion service
> for
> > > various URNs it knows and cares about is probably a pretty low
> bar.
> > > Flink could serve up things it would rather get back untouched in
> a
> > > transform with a special flink runner urn.
> > >
> > > As Ahmet mentions, SDF is better solution. I hope it's not that
> far
> > > away, but even once it comes we'll likely want the above
> > framework to
> > > invoke the full suite of Java IOs even after they're running on
> SDF
> > > themselves.
> > >
> > > - Robert
> > >
> > > On 31.01.19 17:36, Thomas Weise wrote:
> > > > Exactly, that's what I had in mind.
> > > >
> > > > A Flink runner native transform would make the existing
> > unbounded
> > > sources
> > > > available, similar to:
> > > >
> > > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > Wouldn't it be even more useful for the transition
> > period if
> > > we enabled Beam IO
> > > > to be used via Flink (like in the legacy Flink
> Runner)? In
> > > this particular
> > > > example, Matthias wants to use PubSubIO, which is not
> even
> > > available as a
> > > > native
> > > > Flink transform.
> > > >
> > > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > > Until SDF is supported, we could also add Flink
> runner
> > > native transforms for
> > > > > selected unbounded sources [1].
> > > > >
> > > > > That might be a reasonable option to unblock users
> that
> > > want to try Python
> > > > > streaming on Flink.
> > > > >
> > > > > Thomas
> > > > >
> > > > > [1]
> > > > >
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > > >
> > > > >
> > > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >
> > > > > > I have a hard time to imagine how can we map
> > in a
> > > generic way
> > > > > RestrictionTrackers into the existing
> > > Bounded/UnboundedSource, so I would
> > > > > love to hear more about the details.
> > > > >
> > > > > Isn't it the other way around? The SDF is a
> > > generalization of
> > > > UnboundedSource.
> > > > > So we would wrap UnboundedSource using SDF. I'm
> not
> > > saying it is
> > > > trivial, but
> > > > > SDF offers all the functionality that
> > UnboundedSource
> > > needs.
> > > > >
> > > > > For example, the @GetInitialRestriction method
> > would
> > > call split on the
> > > > > UnboundedSource and the restriction trackers
> would
> > > then be used to
> > > > process the
> > > > > splits.
> > > > >
> > > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > > >> Not necessarily. This would be one way.
> Another
> > > way is build an SDF
> > > > > wrapper for UnboundedSource. Probably the
> > easier path
> > > for migration.
> > > > > >
> > > > > > That would be fantastic, I have heard about
> such
> > > wrapper multiple
> > > > > > times but so far there is not any realistic
> > > proposal. I have a hard
> > > > > > time to imagine how can we map in a generic
> way
> > > RestrictionTrackers
> > > > > > into the existing Bounded/UnboundedSource,
> so I
> > > would love to hear
> > > > > > more about the details.
> > > > > >
> > > > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian
> > Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > > >>
> > > > > >> > In addition to have support in the
> runners,
> > > this will require a
> > > > > >> > rewrite of PubsubIO to use the new SDF
> API.
> > > > > >>
> > > > > >> Not necessarily. This would be one way.
> Another
> > > way is build an SDF
> > > > > wrapper for
> > > > > >> UnboundedSource. Probably the easier path
> for
> > > migration.
> > > > > >>
> > > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > > >>>> Fortunately, there is already a pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like PubSub
> in
> > > Python jobs.
> > > > > >>>
> > > > > >>> In addition to have support in the
> > runners, this
> > > will require a
> > > > > >>> rewrite of PubsubIO to use the new SDF API.
> > > > > >>>
> > > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> > Maximilian Michels
> > > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > > >>>>
> > > > > >>>> Hi Matthias,
> > > > > >>>>
> > > > > >>>> This is already reflected in the
> > compatibility
> > > matrix, if you look
> > > > > under SDF.
> > > > > >>>> There is no UnboundedSource interface for
> > > portable pipelines.
> > > > That's a
> > > > > legacy
> > > > > >>>> abstraction that will be replaced with
> SDF.
> > > > > >>>>
> > > > > >>>> Fortunately, there is already a pending
> > PR for
> > > cross-language
> > > > > pipelines which
> > > > > >>>> will allow us to use Java IO like PubSub
> in
> > > Python jobs.
> > > > > >>>>
> > > > > >>>> Thanks,
> > > > > >>>> Max
> > > > > >>>>
> > > > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > > > >>>>> Hey Ankur,
> > > > > >>>>>
> > > > > >>>>> Thanks for the swift reply. Should I
> change
> > > this in the
> > > > capability matrix
> > > > > >>>>>
> > > <https://s.apache.org/apache-beam-portability-support-table>
> > then?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur
> Goenka
> > > <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > > <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>
> > > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi Matthias,
> > > > > >>>>>
> > > > > >>>>> Unfortunately, unbounded reads
> > including
> > > pubsub are not yet
> > > > > supported for
> > > > > >>>>> portable runners.
> > > > > >>>>>
> > > > > >>>>> Thanks,
> > > > > >>>>> Ankur
> > > > > >>>>>
> > > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> > Matthias
> > > Baetens
> > > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>
> > > > > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>>> wrote:
> > > > > >>>>>
> > > > > >>>>> Hi everyone,
> > > > > >>>>>
> > > > > >>>>> Last few days I have been
> > trying to
> > > run a streaming
> > > > > pipeline (code on
> > > > > >>>>> Github
> > > <https://github.com/matthiasa4/beam-demo>) on a
> > > > > Flink Runner.
> > > > > >>>>>
> > > > > >>>>> I am running a Flink cluster
> > locally
> > > (v1.5.6
> > > > > >>>>>
> > > <https://flink.apache.org/downloads.html>)
> > > > > >>>>> I have built the SDK Harness
> > > Container: /./gradlew
> > > > > >>>>>
> > :beam-sdks-python-container:docker/
> > > > > >>>>> and started the JobServer:
> > /./gradlew
> > > > > >>>>>
> > > :beam-runners-flink_2.11-job-server:runShadow
> > > > > >>>>>
> -PflinkMasterUrl=localhost:8081./
> > > > > >>>>>
> > > > > >>>>> I run my pipeline with:
> > /env/bin/python
> > > > streaming_pipeline.py
> > > > > >>>>> --runner=PortableRunner
> > > --job_endpoint=localhost:8099
> > > > > --output xxx
> > > > > >>>>> --input_subscription xxx
> > > --output_subscription xxx/
> > > > > >>>>> /
> > > > > >>>>> /
> > > > > >>>>> All this is running inside a
> > Ubuntu
> > > (Bionic) in a
> > > > Virtualbox.
> > > > > >>>>>
> > > > > >>>>> The job submits fine, but
> > > unfortunately fails after
> > > > a few
> > > > > seconds with
> > > > > >>>>> the error attached.
> > > > > >>>>>
> > > > > >>>>> Anything I am missing or doing
> > wrong?
> > > > > >>>>>
> > > > > >>>>> Many thanks.
> > > > > >>>>> Best,
> > > > > >>>>> Matthias
> > > > > >>>>>
> > > > > >>>>>
> > > > >
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Yes, I imagine sources to implement a JsonConfigurable interface (e.g.
on their builders):
JsonConfigurable {
// Either a json string or Map<String, String>
apply(String jsonConfig);
}
In Python we would create this transform:
URN: JsonConfiguredSource:v1
payload: {
environment: environment_id, // Java/Python/Go
resourceIdentifier: string, // "org.apache.beam.io.PubSubIO"
configuration: json config, // { "topic" : "my_pubsub_topic" }
}
That's more generic and could be used for other languages where we might
have sources/sinks.
> (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi protos that fully specified it, and we use that same format to translate back out.)
Not that I know of.
On 01.02.19 14:02, Robert Bradshaw wrote:
> Are you suggesting something akin to a generic
>
> urn: JsonConfiguredJavaSource
> payload: some json specifying which source and which parameters
>
> which would expand to actually constructing and applying that source?
>
> (FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
> protos that fully specified it, and we use that same format to translate
> back out.)
>
> On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Recaping here:
>
> We all agree that SDF is the way to go for future implementations of
> sources. It enables us to get rid of the source interfaces. However,
> SDF
> does not solve the lack of streaming sources in Python.
>
> The expansion PR (thanks btw!) solves the problem of
> expanding/translating URNs known to an ExpansionService. That is a more
> programmatic way of replacing language-specific performs, instead of
> relying on translators directly in the Runner.
>
> What is unsolved is the configuration of sources from a foreign
> environment. In my opinion this is the most pressing issue for Python
> sources, because what is PubSubIO worth in Python if you cannot
> configure it?
>
> What about this:
>
> I think it is worth adding a JSON configuration option for all existing
> Java sources. That way, we could easily configure them as part of the
> expansion request (which would contain a JSON configuration). I'll
> probably fork a thread to discuss this in more detail, but would
> like to
> hear your thoughts.
>
> -Max
>
> On 01.02.19 13:08, Robert Bradshaw wrote:
> > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Ah, I thought you meant native Flink transforms.
> >
> > Exactly! The translation code is already there. The main
> challenge
> > is how to
> > programmatically configure the BeamIO from Python. I suppose
> that is
> > also an
> > unsolved problem for cross-language transforms in general.
> >
> >
> > This is what https://github.com/apache/beam/pull/7316 does.
> >
> > For a particular source, one would want to define a URN and
> > corresponding payload, then (probably) a CompositeTransform in
> Python
> > that takes the users arguments, packages them into the payload,
> applies
> > the ExternalTransform, and returns the results. How to handle
> arbitrary
> > UDFs embedded in sources is still TBD.
> >
> > For Matthias' pipeline with PubSubIO we can build something
> > specific, but for
> > the general case there should be way to initialize a Beam IO
> via a
> > configuration
> > map provided by an external environment.
> >
> >
> > I thought quite a bit about how we could represent expansions
> statically
> > (e.g. have some kind of expansion template that could be used, at
> least
> > in many cases, as data without firing up a separate process. May be
> > worth doing eventually, but we run into the same issues that were
> > discussed at
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> >
> > If one is already using a portable runner like Flink, having the job
> > service process automatically also serve up an expansion service for
> > various URNs it knows and cares about is probably a pretty low bar.
> > Flink could serve up things it would rather get back untouched in a
> > transform with a special flink runner urn.
> >
> > As Ahmet mentions, SDF is better solution. I hope it's not that far
> > away, but even once it comes we'll likely want the above
> framework to
> > invoke the full suite of Java IOs even after they're running on SDF
> > themselves.
> >
> > - Robert
> >
> > On 31.01.19 17:36, Thomas Weise wrote:
> > > Exactly, that's what I had in mind.
> > >
> > > A Flink runner native transform would make the existing
> unbounded
> > sources
> > > available, similar to:
> > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > >
> > >
> > >
> > >
> > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > Wouldn't it be even more useful for the transition
> period if
> > we enabled Beam IO
> > > to be used via Flink (like in the legacy Flink Runner)? In
> > this particular
> > > example, Matthias wants to use PubSubIO, which is not even
> > available as a
> > > native
> > > Flink transform.
> > >
> > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > Until SDF is supported, we could also add Flink runner
> > native transforms for
> > > > selected unbounded sources [1].
> > > >
> > > > That might be a reasonable option to unblock users that
> > want to try Python
> > > > streaming on Flink.
> > > >
> > > > Thomas
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > >
> > > >
> > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > >
> > > > > I have a hard time to imagine how can we map
> in a
> > generic way
> > > > RestrictionTrackers into the existing
> > Bounded/UnboundedSource, so I would
> > > > love to hear more about the details.
> > > >
> > > > Isn't it the other way around? The SDF is a
> > generalization of
> > > UnboundedSource.
> > > > So we would wrap UnboundedSource using SDF. I'm not
> > saying it is
> > > trivial, but
> > > > SDF offers all the functionality that
> UnboundedSource
> > needs.
> > > >
> > > > For example, the @GetInitialRestriction method
> would
> > call split on the
> > > > UnboundedSource and the restriction trackers would
> > then be used to
> > > process the
> > > > splits.
> > > >
> > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > >> Not necessarily. This would be one way. Another
> > way is build an SDF
> > > > wrapper for UnboundedSource. Probably the
> easier path
> > for migration.
> > > > >
> > > > > That would be fantastic, I have heard about such
> > wrapper multiple
> > > > > times but so far there is not any realistic
> > proposal. I have a hard
> > > > > time to imagine how can we map in a generic way
> > RestrictionTrackers
> > > > > into the existing Bounded/UnboundedSource, so I
> > would love to hear
> > > > > more about the details.
> > > > >
> > > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian
> Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >>
> > > > >> > In addition to have support in the runners,
> > this will require a
> > > > >> > rewrite of PubsubIO to use the new SDF API.
> > > > >>
> > > > >> Not necessarily. This would be one way. Another
> > way is build an SDF
> > > > wrapper for
> > > > >> UnboundedSource. Probably the easier path for
> > migration.
> > > > >>
> > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > >>>> Fortunately, there is already a pending
> PR for
> > cross-language
> > > > pipelines which
> > > > >>>> will allow us to use Java IO like PubSub in
> > Python jobs.
> > > > >>>
> > > > >>> In addition to have support in the
> runners, this
> > will require a
> > > > >>> rewrite of PubsubIO to use the new SDF API.
> > > > >>>
> > > > >>> On Thu, Jan 31, 2019 at 12:23 PM
> Maximilian Michels
> > > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>>> wrote:
> > > > >>>>
> > > > >>>> Hi Matthias,
> > > > >>>>
> > > > >>>> This is already reflected in the
> compatibility
> > matrix, if you look
> > > > under SDF.
> > > > >>>> There is no UnboundedSource interface for
> > portable pipelines.
> > > That's a
> > > > legacy
> > > > >>>> abstraction that will be replaced with SDF.
> > > > >>>>
> > > > >>>> Fortunately, there is already a pending
> PR for
> > cross-language
> > > > pipelines which
> > > > >>>> will allow us to use Java IO like PubSub in
> > Python jobs.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Max
> > > > >>>>
> > > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > > >>>>> Hey Ankur,
> > > > >>>>>
> > > > >>>>> Thanks for the swift reply. Should I change
> > this in the
> > > capability matrix
> > > > >>>>>
> > <https://s.apache.org/apache-beam-portability-support-table>
> then?
> > > > >>>>>
> > > > >>>>> Many thanks.
> > > > >>>>> Best,
> > > > >>>>> Matthias
> > > > >>>>>
> > > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
> > <goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > > <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>
> > > > >>>>> <mailto:goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>>> wrote:
> > > > >>>>>
> > > > >>>>> Hi Matthias,
> > > > >>>>>
> > > > >>>>> Unfortunately, unbounded reads
> including
> > pubsub are not yet
> > > > supported for
> > > > >>>>> portable runners.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Ankur
> > > > >>>>>
> > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM
> Matthias
> > Baetens
> > > > <baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>
> > > > >>>>> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>>> wrote:
> > > > >>>>>
> > > > >>>>> Hi everyone,
> > > > >>>>>
> > > > >>>>> Last few days I have been
> trying to
> > run a streaming
> > > > pipeline (code on
> > > > >>>>> Github
> > <https://github.com/matthiasa4/beam-demo>) on a
> > > > Flink Runner.
> > > > >>>>>
> > > > >>>>> I am running a Flink cluster
> locally
> > (v1.5.6
> > > > >>>>>
> > <https://flink.apache.org/downloads.html>)
> > > > >>>>> I have built the SDK Harness
> > Container: /./gradlew
> > > > >>>>>
> :beam-sdks-python-container:docker/
> > > > >>>>> and started the JobServer:
> /./gradlew
> > > > >>>>>
> > :beam-runners-flink_2.11-job-server:runShadow
> > > > >>>>> -PflinkMasterUrl=localhost:8081./
> > > > >>>>>
> > > > >>>>> I run my pipeline with:
> /env/bin/python
> > > streaming_pipeline.py
> > > > >>>>> --runner=PortableRunner
> > --job_endpoint=localhost:8099
> > > > --output xxx
> > > > >>>>> --input_subscription xxx
> > --output_subscription xxx/
> > > > >>>>> /
> > > > >>>>> /
> > > > >>>>> All this is running inside a
> Ubuntu
> > (Bionic) in a
> > > Virtualbox.
> > > > >>>>>
> > > > >>>>> The job submits fine, but
> > unfortunately fails after
> > > a few
> > > > seconds with
> > > > >>>>> the error attached.
> > > > >>>>>
> > > > >>>>> Anything I am missing or doing
> wrong?
> > > > >>>>>
> > > > >>>>> Many thanks.
> > > > >>>>> Best,
> > > > >>>>> Matthias
> > > > >>>>>
> > > > >>>>>
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
Are you suggesting something akin to a generic
urn: JsonConfiguredJavaSource
payload: some json specifying which source and which parameters
which would expand to actually constructing and applying that source?
(FWIW, I was imagining PubSubIO already had a translation into BeamFnApi
protos that fully specified it, and we use that same format to translate
back out.)
On Fri, Feb 1, 2019 at 1:44 PM Maximilian Michels <mx...@apache.org> wrote:
> Recaping here:
>
> We all agree that SDF is the way to go for future implementations of
> sources. It enables us to get rid of the source interfaces. However, SDF
> does not solve the lack of streaming sources in Python.
>
> The expansion PR (thanks btw!) solves the problem of
> expanding/translating URNs known to an ExpansionService. That is a more
> programmatic way of replacing language-specific performs, instead of
> relying on translators directly in the Runner.
>
> What is unsolved is the configuration of sources from a foreign
> environment. In my opinion this is the most pressing issue for Python
> sources, because what is PubSubIO worth in Python if you cannot
> configure it?
>
> What about this:
>
> I think it is worth adding a JSON configuration option for all existing
> Java sources. That way, we could easily configure them as part of the
> expansion request (which would contain a JSON configuration). I'll
> probably fork a thread to discuss this in more detail, but would like to
> hear your thoughts.
>
> -Max
>
> On 01.02.19 13:08, Robert Bradshaw wrote:
> > On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Ah, I thought you meant native Flink transforms.
> >
> > Exactly! The translation code is already there. The main challenge
> > is how to
> > programmatically configure the BeamIO from Python. I suppose that is
> > also an
> > unsolved problem for cross-language transforms in general.
> >
> >
> > This is what https://github.com/apache/beam/pull/7316 does.
> >
> > For a particular source, one would want to define a URN and
> > corresponding payload, then (probably) a CompositeTransform in Python
> > that takes the users arguments, packages them into the payload, applies
> > the ExternalTransform, and returns the results. How to handle arbitrary
> > UDFs embedded in sources is still TBD.
> >
> > For Matthias' pipeline with PubSubIO we can build something
> > specific, but for
> > the general case there should be way to initialize a Beam IO via a
> > configuration
> > map provided by an external environment.
> >
> >
> > I thought quite a bit about how we could represent expansions statically
> > (e.g. have some kind of expansion template that could be used, at least
> > in many cases, as data without firing up a separate process. May be
> > worth doing eventually, but we run into the same issues that were
> > discussed at
> > https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
> >
> > If one is already using a portable runner like Flink, having the job
> > service process automatically also serve up an expansion service for
> > various URNs it knows and cares about is probably a pretty low bar.
> > Flink could serve up things it would rather get back untouched in a
> > transform with a special flink runner urn.
> >
> > As Ahmet mentions, SDF is better solution. I hope it's not that far
> > away, but even once it comes we'll likely want the above framework to
> > invoke the full suite of Java IOs even after they're running on SDF
> > themselves.
> >
> > - Robert
> >
> > On 31.01.19 17:36, Thomas Weise wrote:
> > > Exactly, that's what I had in mind.
> > >
> > > A Flink runner native transform would make the existing unbounded
> > sources
> > > available, similar to:
> > >
> > >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> > >
> > >
> > >
> > >
> > > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Wouldn't it be even more useful for the transition period if
> > we enabled Beam IO
> > > to be used via Flink (like in the legacy Flink Runner)? In
> > this particular
> > > example, Matthias wants to use PubSubIO, which is not even
> > available as a
> > > native
> > > Flink transform.
> > >
> > > On 31.01.19 16:21, Thomas Weise wrote:
> > > > Until SDF is supported, we could also add Flink runner
> > native transforms for
> > > > selected unbounded sources [1].
> > > >
> > > > That might be a reasonable option to unblock users that
> > want to try Python
> > > > streaming on Flink.
> > > >
> > > > Thomas
> > > >
> > > > [1]
> > > >
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > > >
> > > >
> > > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >
> > > > > I have a hard time to imagine how can we map in a
> > generic way
> > > > RestrictionTrackers into the existing
> > Bounded/UnboundedSource, so I would
> > > > love to hear more about the details.
> > > >
> > > > Isn't it the other way around? The SDF is a
> > generalization of
> > > UnboundedSource.
> > > > So we would wrap UnboundedSource using SDF. I'm not
> > saying it is
> > > trivial, but
> > > > SDF offers all the functionality that UnboundedSource
> > needs.
> > > >
> > > > For example, the @GetInitialRestriction method would
> > call split on the
> > > > UnboundedSource and the restriction trackers would
> > then be used to
> > > process the
> > > > splits.
> > > >
> > > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > > >> Not necessarily. This would be one way. Another
> > way is build an SDF
> > > > wrapper for UnboundedSource. Probably the easier path
> > for migration.
> > > > >
> > > > > That would be fantastic, I have heard about such
> > wrapper multiple
> > > > > times but so far there is not any realistic
> > proposal. I have a hard
> > > > > time to imagine how can we map in a generic way
> > RestrictionTrackers
> > > > > into the existing Bounded/UnboundedSource, so I
> > would love to hear
> > > > > more about the details.
> > > > >
> > > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > > >>
> > > > >> > In addition to have support in the runners,
> > this will require a
> > > > >> > rewrite of PubsubIO to use the new SDF API.
> > > > >>
> > > > >> Not necessarily. This would be one way. Another
> > way is build an SDF
> > > > wrapper for
> > > > >> UnboundedSource. Probably the easier path for
> > migration.
> > > > >>
> > > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > > >>>> Fortunately, there is already a pending PR for
> > cross-language
> > > > pipelines which
> > > > >>>> will allow us to use Java IO like PubSub in
> > Python jobs.
> > > > >>>
> > > > >>> In addition to have support in the runners, this
> > will require a
> > > > >>> rewrite of PubsubIO to use the new SDF API.
> > > > >>>
> > > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian
> Michels
> > > <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > > <mailto:mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > > >>>>
> > > > >>>> Hi Matthias,
> > > > >>>>
> > > > >>>> This is already reflected in the compatibility
> > matrix, if you look
> > > > under SDF.
> > > > >>>> There is no UnboundedSource interface for
> > portable pipelines.
> > > That's a
> > > > legacy
> > > > >>>> abstraction that will be replaced with SDF.
> > > > >>>>
> > > > >>>> Fortunately, there is already a pending PR for
> > cross-language
> > > > pipelines which
> > > > >>>> will allow us to use Java IO like PubSub in
> > Python jobs.
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>> Max
> > > > >>>>
> > > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > > >>>>> Hey Ankur,
> > > > >>>>>
> > > > >>>>> Thanks for the swift reply. Should I change
> > this in the
> > > capability matrix
> > > > >>>>>
> > <https://s.apache.org/apache-beam-portability-support-table> then?
> > > > >>>>>
> > > > >>>>> Many thanks.
> > > > >>>>> Best,
> > > > >>>>> Matthias
> > > > >>>>>
> > > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
> > <goenka@google.com <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>
> > > > >>>>> <mailto:goenka@google.com
> > <ma...@google.com> <mailto:goenka@google.com
> > <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
> > > > >>>>>
> > > > >>>>> Hi Matthias,
> > > > >>>>>
> > > > >>>>> Unfortunately, unbounded reads including
> > pubsub are not yet
> > > > supported for
> > > > >>>>> portable runners.
> > > > >>>>>
> > > > >>>>> Thanks,
> > > > >>>>> Ankur
> > > > >>>>>
> > > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias
> > Baetens
> > > > <baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>
> > > > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>
> > > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>>> wrote:
> > > > >>>>>
> > > > >>>>> Hi everyone,
> > > > >>>>>
> > > > >>>>> Last few days I have been trying to
> > run a streaming
> > > > pipeline (code on
> > > > >>>>> Github
> > <https://github.com/matthiasa4/beam-demo>) on a
> > > > Flink Runner.
> > > > >>>>>
> > > > >>>>> I am running a Flink cluster locally
> > (v1.5.6
> > > > >>>>>
> > <https://flink.apache.org/downloads.html>)
> > > > >>>>> I have built the SDK Harness
> > Container: /./gradlew
> > > > >>>>> :beam-sdks-python-container:docker/
> > > > >>>>> and started the JobServer: /./gradlew
> > > > >>>>>
> > :beam-runners-flink_2.11-job-server:runShadow
> > > > >>>>> -PflinkMasterUrl=localhost:8081./
> > > > >>>>>
> > > > >>>>> I run my pipeline with:
> /env/bin/python
> > > streaming_pipeline.py
> > > > >>>>> --runner=PortableRunner
> > --job_endpoint=localhost:8099
> > > > --output xxx
> > > > >>>>> --input_subscription xxx
> > --output_subscription xxx/
> > > > >>>>> /
> > > > >>>>> /
> > > > >>>>> All this is running inside a Ubuntu
> > (Bionic) in a
> > > Virtualbox.
> > > > >>>>>
> > > > >>>>> The job submits fine, but
> > unfortunately fails after
> > > a few
> > > > seconds with
> > > > >>>>> the error attached.
> > > > >>>>>
> > > > >>>>> Anything I am missing or doing wrong?
> > > > >>>>>
> > > > >>>>> Many thanks.
> > > > >>>>> Best,
> > > > >>>>> Matthias
> > > > >>>>>
> > > > >>>>>
> > > >
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Recaping here:
We all agree that SDF is the way to go for future implementations of
sources. It enables us to get rid of the source interfaces. However, SDF
does not solve the lack of streaming sources in Python.
The expansion PR (thanks btw!) solves the problem of
expanding/translating URNs known to an ExpansionService. That is a more
programmatic way of replacing language-specific performs, instead of
relying on translators directly in the Runner.
What is unsolved is the configuration of sources from a foreign
environment. In my opinion this is the most pressing issue for Python
sources, because what is PubSubIO worth in Python if you cannot
configure it?
What about this:
I think it is worth adding a JSON configuration option for all existing
Java sources. That way, we could easily configure them as part of the
expansion request (which would contain a JSON configuration). I'll
probably fork a thread to discuss this in more detail, but would like to
hear your thoughts.
-Max
On 01.02.19 13:08, Robert Bradshaw wrote:
> On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Ah, I thought you meant native Flink transforms.
>
> Exactly! The translation code is already there. The main challenge
> is how to
> programmatically configure the BeamIO from Python. I suppose that is
> also an
> unsolved problem for cross-language transforms in general.
>
>
> This is what https://github.com/apache/beam/pull/7316 does.
>
> For a particular source, one would want to define a URN and
> corresponding payload, then (probably) a CompositeTransform in Python
> that takes the users arguments, packages them into the payload, applies
> the ExternalTransform, and returns the results. How to handle arbitrary
> UDFs embedded in sources is still TBD.
>
> For Matthias' pipeline with PubSubIO we can build something
> specific, but for
> the general case there should be way to initialize a Beam IO via a
> configuration
> map provided by an external environment.
>
>
> I thought quite a bit about how we could represent expansions statically
> (e.g. have some kind of expansion template that could be used, at least
> in many cases, as data without firing up a separate process. May be
> worth doing eventually, but we run into the same issues that were
> discussed at
> https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
>
> If one is already using a portable runner like Flink, having the job
> service process automatically also serve up an expansion service for
> various URNs it knows and cares about is probably a pretty low bar.
> Flink could serve up things it would rather get back untouched in a
> transform with a special flink runner urn.
>
> As Ahmet mentions, SDF is better solution. I hope it's not that far
> away, but even once it comes we'll likely want the above framework to
> invoke the full suite of Java IOs even after they're running on SDF
> themselves.
>
> - Robert
>
> On 31.01.19 17:36, Thomas Weise wrote:
> > Exactly, that's what I had in mind.
> >
> > A Flink runner native transform would make the existing unbounded
> sources
> > available, similar to:
> >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> >
> >
> >
> >
> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Wouldn't it be even more useful for the transition period if
> we enabled Beam IO
> > to be used via Flink (like in the legacy Flink Runner)? In
> this particular
> > example, Matthias wants to use PubSubIO, which is not even
> available as a
> > native
> > Flink transform.
> >
> > On 31.01.19 16:21, Thomas Weise wrote:
> > > Until SDF is supported, we could also add Flink runner
> native transforms for
> > > selected unbounded sources [1].
> > >
> > > That might be a reasonable option to unblock users that
> want to try Python
> > > streaming on Flink.
> > >
> > > Thomas
> > >
> > > [1]
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > >
> > >
> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > >
> > > > I have a hard time to imagine how can we map in a
> generic way
> > > RestrictionTrackers into the existing
> Bounded/UnboundedSource, so I would
> > > love to hear more about the details.
> > >
> > > Isn't it the other way around? The SDF is a
> generalization of
> > UnboundedSource.
> > > So we would wrap UnboundedSource using SDF. I'm not
> saying it is
> > trivial, but
> > > SDF offers all the functionality that UnboundedSource
> needs.
> > >
> > > For example, the @GetInitialRestriction method would
> call split on the
> > > UnboundedSource and the restriction trackers would
> then be used to
> > process the
> > > splits.
> > >
> > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > >> Not necessarily. This would be one way. Another
> way is build an SDF
> > > wrapper for UnboundedSource. Probably the easier path
> for migration.
> > > >
> > > > That would be fantastic, I have heard about such
> wrapper multiple
> > > > times but so far there is not any realistic
> proposal. I have a hard
> > > > time to imagine how can we map in a generic way
> RestrictionTrackers
> > > > into the existing Bounded/UnboundedSource, so I
> would love to hear
> > > > more about the details.
> > > >
> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >>
> > > >> > In addition to have support in the runners,
> this will require a
> > > >> > rewrite of PubsubIO to use the new SDF API.
> > > >>
> > > >> Not necessarily. This would be one way. Another
> way is build an SDF
> > > wrapper for
> > > >> UnboundedSource. Probably the easier path for
> migration.
> > > >>
> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in
> Python jobs.
> > > >>>
> > > >>> In addition to have support in the runners, this
> will require a
> > > >>> rewrite of PubsubIO to use the new SDF API.
> > > >>>
> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>
> > > <mailto:mxm@apache.org <ma...@apache.org>
> <mailto:mxm@apache.org <ma...@apache.org>>>> wrote:
> > > >>>>
> > > >>>> Hi Matthias,
> > > >>>>
> > > >>>> This is already reflected in the compatibility
> matrix, if you look
> > > under SDF.
> > > >>>> There is no UnboundedSource interface for
> portable pipelines.
> > That's a
> > > legacy
> > > >>>> abstraction that will be replaced with SDF.
> > > >>>>
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in
> Python jobs.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Max
> > > >>>>
> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > >>>>> Hey Ankur,
> > > >>>>>
> > > >>>>> Thanks for the swift reply. Should I change
> this in the
> > capability matrix
> > > >>>>>
> <https://s.apache.org/apache-beam-portability-support-table> then?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka
> <goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>
> > > >>>>> <mailto:goenka@google.com
> <ma...@google.com> <mailto:goenka@google.com
> <ma...@google.com>>
> > <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>>> wrote:
> > > >>>>>
> > > >>>>> Hi Matthias,
> > > >>>>>
> > > >>>>> Unfortunately, unbounded reads including
> pubsub are not yet
> > > supported for
> > > >>>>> portable runners.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Ankur
> > > >>>>>
> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias
> Baetens
> > > <baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>
> > > >>>>> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>
> > > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>>> wrote:
> > > >>>>>
> > > >>>>> Hi everyone,
> > > >>>>>
> > > >>>>> Last few days I have been trying to
> run a streaming
> > > pipeline (code on
> > > >>>>> Github
> <https://github.com/matthiasa4/beam-demo>) on a
> > > Flink Runner.
> > > >>>>>
> > > >>>>> I am running a Flink cluster locally
> (v1.5.6
> > > >>>>>
> <https://flink.apache.org/downloads.html>)
> > > >>>>> I have built the SDK Harness
> Container: /./gradlew
> > > >>>>> :beam-sdks-python-container:docker/
> > > >>>>> and started the JobServer: /./gradlew
> > > >>>>>
> :beam-runners-flink_2.11-job-server:runShadow
> > > >>>>> -PflinkMasterUrl=localhost:8081./
> > > >>>>>
> > > >>>>> I run my pipeline with: /env/bin/python
> > streaming_pipeline.py
> > > >>>>> --runner=PortableRunner
> --job_endpoint=localhost:8099
> > > --output xxx
> > > >>>>> --input_subscription xxx
> --output_subscription xxx/
> > > >>>>> /
> > > >>>>> /
> > > >>>>> All this is running inside a Ubuntu
> (Bionic) in a
> > Virtualbox.
> > > >>>>>
> > > >>>>> The job submits fine, but
> unfortunately fails after
> > a few
> > > seconds with
> > > >>>>> the error attached.
> > > >>>>>
> > > >>>>> Anything I am missing or doing wrong?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>>
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, Jan 31, 2019 at 6:25 PM Maximilian Michels <mx...@apache.org> wrote:
> Ah, I thought you meant native Flink transforms.
>
> Exactly! The translation code is already there. The main challenge is how
> to
> programmatically configure the BeamIO from Python. I suppose that is also
> an
> unsolved problem for cross-language transforms in general.
>
This is what https://github.com/apache/beam/pull/7316 does.
For a particular source, one would want to define a URN and corresponding
payload, then (probably) a CompositeTransform in Python that takes the
users arguments, packages them into the payload, applies the
ExternalTransform, and returns the results. How to handle arbitrary UDFs
embedded in sources is still TBD.
> For Matthias' pipeline with PubSubIO we can build something specific, but
> for
> the general case there should be way to initialize a Beam IO via a
> configuration
> map provided by an external environment.
>
I thought quite a bit about how we could represent expansions statically
(e.g. have some kind of expansion template that could be used, at least in
many cases, as data without firing up a separate process. May be worth
doing eventually, but we run into the same issues that were discussed at
https://github.com/apache/beam/pull/7316#discussion_r249996455 ).
If one is already using a portable runner like Flink, having the job
service process automatically also serve up an expansion service for
various URNs it knows and cares about is probably a pretty low bar. Flink
could serve up things it would rather get back untouched in a transform
with a special flink runner urn.
As Ahmet mentions, SDF is better solution. I hope it's not that far away,
but even once it comes we'll likely want the above framework to invoke the
full suite of Java IOs even after they're running on SDF themselves.
- Robert
> On 31.01.19 17:36, Thomas Weise wrote:
> > Exactly, that's what I had in mind.
> >
> > A Flink runner native transform would make the existing unbounded
> sources
> > available, similar to:
> >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> >
> >
> >
> >
> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Wouldn't it be even more useful for the transition period if we
> enabled Beam IO
> > to be used via Flink (like in the legacy Flink Runner)? In this
> particular
> > example, Matthias wants to use PubSubIO, which is not even available
> as a
> > native
> > Flink transform.
> >
> > On 31.01.19 16:21, Thomas Weise wrote:
> > > Until SDF is supported, we could also add Flink runner native
> transforms for
> > > selected unbounded sources [1].
> > >
> > > That might be a reasonable option to unblock users that want to
> try Python
> > > streaming on Flink.
> > >
> > > Thomas
> > >
> > > [1]
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > >
> > >
> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > > I have a hard time to imagine how can we map in a generic
> way
> > > RestrictionTrackers into the existing
> Bounded/UnboundedSource, so I would
> > > love to hear more about the details.
> > >
> > > Isn't it the other way around? The SDF is a generalization of
> > UnboundedSource.
> > > So we would wrap UnboundedSource using SDF. I'm not saying it
> is
> > trivial, but
> > > SDF offers all the functionality that UnboundedSource needs.
> > >
> > > For example, the @GetInitialRestriction method would call
> split on the
> > > UnboundedSource and the restriction trackers would then be
> used to
> > process the
> > > splits.
> > >
> > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > >> Not necessarily. This would be one way. Another way is
> build an SDF
> > > wrapper for UnboundedSource. Probably the easier path for
> migration.
> > > >
> > > > That would be fantastic, I have heard about such wrapper
> multiple
> > > > times but so far there is not any realistic proposal. I
> have a hard
> > > > time to imagine how can we map in a generic way
> RestrictionTrackers
> > > > into the existing Bounded/UnboundedSource, so I would love
> to hear
> > > > more about the details.
> > > >
> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >>
> > > >> > In addition to have support in the runners, this will
> require a
> > > >> > rewrite of PubsubIO to use the new SDF API.
> > > >>
> > > >> Not necessarily. This would be one way. Another way is
> build an SDF
> > > wrapper for
> > > >> UnboundedSource. Probably the easier path for migration.
> > > >>
> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > > >>>
> > > >>> In addition to have support in the runners, this will
> require a
> > > >>> rewrite of PubsubIO to use the new SDF API.
> > > >>>
> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >>>>
> > > >>>> Hi Matthias,
> > > >>>>
> > > >>>> This is already reflected in the compatibility matrix,
> if you look
> > > under SDF.
> > > >>>> There is no UnboundedSource interface for portable
> pipelines.
> > That's a
> > > legacy
> > > >>>> abstraction that will be replaced with SDF.
> > > >>>>
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Max
> > > >>>>
> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > >>>>> Hey Ankur,
> > > >>>>>
> > > >>>>> Thanks for the swift reply. Should I change this in the
> > capability matrix
> > > >>>>> <
> https://s.apache.org/apache-beam-portability-support-table> then?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <
> goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > >>>>> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>> wrote:
> > > >>>>>
> > > >>>>> Hi Matthias,
> > > >>>>>
> > > >>>>> Unfortunately, unbounded reads including pubsub
> are not yet
> > > supported for
> > > >>>>> portable runners.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Ankur
> > > >>>>>
> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens
> > > <baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com <mailto:baetensmatthias@gmail.com
> >>
> > > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>> wrote:
> > > >>>>>
> > > >>>>> Hi everyone,
> > > >>>>>
> > > >>>>> Last few days I have been trying to run a
> streaming
> > > pipeline (code on
> > > >>>>> Github <
> https://github.com/matthiasa4/beam-demo>) on a
> > > Flink Runner.
> > > >>>>>
> > > >>>>> I am running a Flink cluster locally (v1.5.6
> > > >>>>> <https://flink.apache.org/downloads.html>)
> > > >>>>> I have built the SDK Harness Container:
> /./gradlew
> > > >>>>> :beam-sdks-python-container:docker/
> > > >>>>> and started the JobServer: /./gradlew
> > > >>>>> :beam-runners-flink_2.11-job-server:runShadow
> > > >>>>> -PflinkMasterUrl=localhost:8081./
> > > >>>>>
> > > >>>>> I run my pipeline with: /env/bin/python
> > streaming_pipeline.py
> > > >>>>> --runner=PortableRunner
> --job_endpoint=localhost:8099
> > > --output xxx
> > > >>>>> --input_subscription xxx
> --output_subscription xxx/
> > > >>>>> /
> > > >>>>> /
> > > >>>>> All this is running inside a Ubuntu (Bionic)
> in a
> > Virtualbox.
> > > >>>>>
> > > >>>>> The job submits fine, but unfortunately
> fails after
> > a few
> > > seconds with
> > > >>>>> the error attached.
> > > >>>>>
> > > >>>>> Anything I am missing or doing wrong?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>>
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ahmet Altay <al...@google.com>.
+1 to Thomas's idea as a way to enable python users on Flink. On the other
hand his will be a throwaway work once SDF is supported. How far are we
from SDF support?
On Thu, Jan 31, 2019 at 9:18 AM Maximilian Michels <mx...@apache.org> wrote:
> Ah, I thought you meant native Flink transforms.
>
> Exactly! The translation code is already there. The main challenge is how
> to
> programmatically configure the BeamIO from Python. I suppose that is also
> an
> unsolved problem for cross-language transforms in general.
>
> For Matthias' pipeline with PubSubIO we can build something specific, but
> for
> the general case there should be way to initialize a Beam IO via a
> configuration
> map provided by an external environment.
>
> On 31.01.19 17:36, Thomas Weise wrote:
> > Exactly, that's what I had in mind.
> >
> > A Flink runner native transform would make the existing unbounded
> sources
> > available, similar to:
> >
> >
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
> >
> >
> >
> >
> > On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Wouldn't it be even more useful for the transition period if we
> enabled Beam IO
> > to be used via Flink (like in the legacy Flink Runner)? In this
> particular
> > example, Matthias wants to use PubSubIO, which is not even available
> as a
> > native
> > Flink transform.
> >
> > On 31.01.19 16:21, Thomas Weise wrote:
> > > Until SDF is supported, we could also add Flink runner native
> transforms for
> > > selected unbounded sources [1].
> > >
> > > That might be a reasonable option to unblock users that want to
> try Python
> > > streaming on Flink.
> > >
> > > Thomas
> > >
> > > [1]
> > >
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> > >
> > >
> > > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > > I have a hard time to imagine how can we map in a generic
> way
> > > RestrictionTrackers into the existing
> Bounded/UnboundedSource, so I would
> > > love to hear more about the details.
> > >
> > > Isn't it the other way around? The SDF is a generalization of
> > UnboundedSource.
> > > So we would wrap UnboundedSource using SDF. I'm not saying it
> is
> > trivial, but
> > > SDF offers all the functionality that UnboundedSource needs.
> > >
> > > For example, the @GetInitialRestriction method would call
> split on the
> > > UnboundedSource and the restriction trackers would then be
> used to
> > process the
> > > splits.
> > >
> > > On 31.01.19 15:16, Ismaël Mejía wrote:
> > > >> Not necessarily. This would be one way. Another way is
> build an SDF
> > > wrapper for UnboundedSource. Probably the easier path for
> migration.
> > > >
> > > > That would be fantastic, I have heard about such wrapper
> multiple
> > > > times but so far there is not any realistic proposal. I
> have a hard
> > > > time to imagine how can we map in a generic way
> RestrictionTrackers
> > > > into the existing Bounded/UnboundedSource, so I would love
> to hear
> > > > more about the details.
> > > >
> > > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >>
> > > >> > In addition to have support in the runners, this will
> require a
> > > >> > rewrite of PubsubIO to use the new SDF API.
> > > >>
> > > >> Not necessarily. This would be one way. Another way is
> build an SDF
> > > wrapper for
> > > >> UnboundedSource. Probably the easier path for migration.
> > > >>
> > > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > > >>>
> > > >>> In addition to have support in the runners, this will
> require a
> > > >>> rewrite of PubsubIO to use the new SDF API.
> > > >>>
> > > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > > >>>>
> > > >>>> Hi Matthias,
> > > >>>>
> > > >>>> This is already reflected in the compatibility matrix,
> if you look
> > > under SDF.
> > > >>>> There is no UnboundedSource interface for portable
> pipelines.
> > That's a
> > > legacy
> > > >>>> abstraction that will be replaced with SDF.
> > > >>>>
> > > >>>> Fortunately, there is already a pending PR for
> cross-language
> > > pipelines which
> > > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Max
> > > >>>>
> > > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > > >>>>> Hey Ankur,
> > > >>>>>
> > > >>>>> Thanks for the swift reply. Should I change this in the
> > capability matrix
> > > >>>>> <
> https://s.apache.org/apache-beam-portability-support-table> then?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <
> goenka@google.com
> > <ma...@google.com>
> > > <mailto:goenka@google.com <ma...@google.com>>
> > > >>>>> <mailto:goenka@google.com <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>>> wrote:
> > > >>>>>
> > > >>>>> Hi Matthias,
> > > >>>>>
> > > >>>>> Unfortunately, unbounded reads including pubsub
> are not yet
> > > supported for
> > > >>>>> portable runners.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Ankur
> > > >>>>>
> > > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens
> > > <baetensmatthias@gmail.com <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com <mailto:baetensmatthias@gmail.com
> >>
> > > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>
> > > <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>>> wrote:
> > > >>>>>
> > > >>>>> Hi everyone,
> > > >>>>>
> > > >>>>> Last few days I have been trying to run a
> streaming
> > > pipeline (code on
> > > >>>>> Github <
> https://github.com/matthiasa4/beam-demo>) on a
> > > Flink Runner.
> > > >>>>>
> > > >>>>> I am running a Flink cluster locally (v1.5.6
> > > >>>>> <https://flink.apache.org/downloads.html>)
> > > >>>>> I have built the SDK Harness Container:
> /./gradlew
> > > >>>>> :beam-sdks-python-container:docker/
> > > >>>>> and started the JobServer: /./gradlew
> > > >>>>> :beam-runners-flink_2.11-job-server:runShadow
> > > >>>>> -PflinkMasterUrl=localhost:8081./
> > > >>>>>
> > > >>>>> I run my pipeline with: /env/bin/python
> > streaming_pipeline.py
> > > >>>>> --runner=PortableRunner
> --job_endpoint=localhost:8099
> > > --output xxx
> > > >>>>> --input_subscription xxx
> --output_subscription xxx/
> > > >>>>> /
> > > >>>>> /
> > > >>>>> All this is running inside a Ubuntu (Bionic)
> in a
> > Virtualbox.
> > > >>>>>
> > > >>>>> The job submits fine, but unfortunately
> fails after
> > a few
> > > seconds with
> > > >>>>> the error attached.
> > > >>>>>
> > > >>>>> Anything I am missing or doing wrong?
> > > >>>>>
> > > >>>>> Many thanks.
> > > >>>>> Best,
> > > >>>>> Matthias
> > > >>>>>
> > > >>>>>
> > >
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Ah, I thought you meant native Flink transforms.
Exactly! The translation code is already there. The main challenge is how to
programmatically configure the BeamIO from Python. I suppose that is also an
unsolved problem for cross-language transforms in general.
For Matthias' pipeline with PubSubIO we can build something specific, but for
the general case there should be way to initialize a Beam IO via a configuration
map provided by an external environment.
On 31.01.19 17:36, Thomas Weise wrote:
> Exactly, that's what I had in mind.
>
> A Flink runner native transform would make the existing unbounded sources
> available, similar to:
>
> https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
>
>
>
>
> On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Wouldn't it be even more useful for the transition period if we enabled Beam IO
> to be used via Flink (like in the legacy Flink Runner)? In this particular
> example, Matthias wants to use PubSubIO, which is not even available as a
> native
> Flink transform.
>
> On 31.01.19 16:21, Thomas Weise wrote:
> > Until SDF is supported, we could also add Flink runner native transforms for
> > selected unbounded sources [1].
> >
> > That might be a reasonable option to unblock users that want to try Python
> > streaming on Flink.
> >
> > Thomas
> >
> > [1]
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> >
> >
> > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > > I have a hard time to imagine how can we map in a generic way
> > RestrictionTrackers into the existing Bounded/UnboundedSource, so I would
> > love to hear more about the details.
> >
> > Isn't it the other way around? The SDF is a generalization of
> UnboundedSource.
> > So we would wrap UnboundedSource using SDF. I'm not saying it is
> trivial, but
> > SDF offers all the functionality that UnboundedSource needs.
> >
> > For example, the @GetInitialRestriction method would call split on the
> > UnboundedSource and the restriction trackers would then be used to
> process the
> > splits.
> >
> > On 31.01.19 15:16, Ismaël Mejía wrote:
> > >> Not necessarily. This would be one way. Another way is build an SDF
> > wrapper for UnboundedSource. Probably the easier path for migration.
> > >
> > > That would be fantastic, I have heard about such wrapper multiple
> > > times but so far there is not any realistic proposal. I have a hard
> > > time to imagine how can we map in a generic way RestrictionTrackers
> > > into the existing Bounded/UnboundedSource, so I would love to hear
> > > more about the details.
> > >
> > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >>
> > >> > In addition to have support in the runners, this will require a
> > >> > rewrite of PubsubIO to use the new SDF API.
> > >>
> > >> Not necessarily. This would be one way. Another way is build an SDF
> > wrapper for
> > >> UnboundedSource. Probably the easier path for migration.
> > >>
> > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > >>>> Fortunately, there is already a pending PR for cross-language
> > pipelines which
> > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > >>>
> > >>> In addition to have support in the runners, this will require a
> > >>> rewrite of PubsubIO to use the new SDF API.
> > >>>
> > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >>>>
> > >>>> Hi Matthias,
> > >>>>
> > >>>> This is already reflected in the compatibility matrix, if you look
> > under SDF.
> > >>>> There is no UnboundedSource interface for portable pipelines.
> That's a
> > legacy
> > >>>> abstraction that will be replaced with SDF.
> > >>>>
> > >>>> Fortunately, there is already a pending PR for cross-language
> > pipelines which
> > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > >>>>
> > >>>> Thanks,
> > >>>> Max
> > >>>>
> > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > >>>>> Hey Ankur,
> > >>>>>
> > >>>>> Thanks for the swift reply. Should I change this in the
> capability matrix
> > >>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
> > >>>>>
> > >>>>> Many thanks.
> > >>>>> Best,
> > >>>>> Matthias
> > >>>>>
> > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> <ma...@google.com>
> > <mailto:goenka@google.com <ma...@google.com>>
> > >>>>> <mailto:goenka@google.com <ma...@google.com>
> <mailto:goenka@google.com <ma...@google.com>>>> wrote:
> > >>>>>
> > >>>>> Hi Matthias,
> > >>>>>
> > >>>>> Unfortunately, unbounded reads including pubsub are not yet
> > supported for
> > >>>>> portable runners.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Ankur
> > >>>>>
> > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens
> > <baetensmatthias@gmail.com <ma...@gmail.com>
> <mailto:baetensmatthias@gmail.com <ma...@gmail.com>>
> > >>>>> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>
> > <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>>> wrote:
> > >>>>>
> > >>>>> Hi everyone,
> > >>>>>
> > >>>>> Last few days I have been trying to run a streaming
> > pipeline (code on
> > >>>>> Github <https://github.com/matthiasa4/beam-demo>) on a
> > Flink Runner.
> > >>>>>
> > >>>>> I am running a Flink cluster locally (v1.5.6
> > >>>>> <https://flink.apache.org/downloads.html>)
> > >>>>> I have built the SDK Harness Container: /./gradlew
> > >>>>> :beam-sdks-python-container:docker/
> > >>>>> and started the JobServer: /./gradlew
> > >>>>> :beam-runners-flink_2.11-job-server:runShadow
> > >>>>> -PflinkMasterUrl=localhost:8081./
> > >>>>>
> > >>>>> I run my pipeline with: /env/bin/python
> streaming_pipeline.py
> > >>>>> --runner=PortableRunner --job_endpoint=localhost:8099
> > --output xxx
> > >>>>> --input_subscription xxx --output_subscription xxx/
> > >>>>> /
> > >>>>> /
> > >>>>> All this is running inside a Ubuntu (Bionic) in a
> Virtualbox.
> > >>>>>
> > >>>>> The job submits fine, but unfortunately fails after
> a few
> > seconds with
> > >>>>> the error attached.
> > >>>>>
> > >>>>> Anything I am missing or doing wrong?
> > >>>>>
> > >>>>> Many thanks.
> > >>>>> Best,
> > >>>>> Matthias
> > >>>>>
> > >>>>>
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Thomas Weise <th...@apache.org>.
Exactly, that's what I had in mind.
A Flink runner native transform would make the existing unbounded sources
available, similar to:
https://github.com/apache/beam/blob/2e89c1e4d35e7b5f95a622259d23d921c3d6ad1f/runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStreamingTransformTranslators.java#L167
On Thu, Jan 31, 2019 at 8:18 AM Maximilian Michels <mx...@apache.org> wrote:
> Wouldn't it be even more useful for the transition period if we enabled
> Beam IO
> to be used via Flink (like in the legacy Flink Runner)? In this particular
> example, Matthias wants to use PubSubIO, which is not even available as a
> native
> Flink transform.
>
> On 31.01.19 16:21, Thomas Weise wrote:
> > Until SDF is supported, we could also add Flink runner native transforms
> for
> > selected unbounded sources [1].
> >
> > That might be a reasonable option to unblock users that want to try
> Python
> > streaming on Flink.
> >
> > Thomas
> >
> > [1]
> >
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
> >
> >
> > On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > > I have a hard time to imagine how can we map in a generic way
> > RestrictionTrackers into the existing Bounded/UnboundedSource, so I
> would
> > love to hear more about the details.
> >
> > Isn't it the other way around? The SDF is a generalization of
> UnboundedSource.
> > So we would wrap UnboundedSource using SDF. I'm not saying it is
> trivial, but
> > SDF offers all the functionality that UnboundedSource needs.
> >
> > For example, the @GetInitialRestriction method would call split on
> the
> > UnboundedSource and the restriction trackers would then be used to
> process the
> > splits.
> >
> > On 31.01.19 15:16, Ismaël Mejía wrote:
> > >> Not necessarily. This would be one way. Another way is build an
> SDF
> > wrapper for UnboundedSource. Probably the easier path for migration.
> > >
> > > That would be fantastic, I have heard about such wrapper multiple
> > > times but so far there is not any realistic proposal. I have a
> hard
> > > time to imagine how can we map in a generic way
> RestrictionTrackers
> > > into the existing Bounded/UnboundedSource, so I would love to hear
> > > more about the details.
> > >
> > > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>> wrote:
> > >>
> > >> > In addition to have support in the runners, this will
> require a
> > >> > rewrite of PubsubIO to use the new SDF API.
> > >>
> > >> Not necessarily. This would be one way. Another way is build an
> SDF
> > wrapper for
> > >> UnboundedSource. Probably the easier path for migration.
> > >>
> > >> On 31.01.19 14:03, Ismaël Mejía wrote:
> > >>>> Fortunately, there is already a pending PR for cross-language
> > pipelines which
> > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > >>>
> > >>> In addition to have support in the runners, this will require a
> > >>> rewrite of PubsubIO to use the new SDF API.
> > >>>
> > >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <
> mxm@apache.org
> > <ma...@apache.org>> wrote:
> > >>>>
> > >>>> Hi Matthias,
> > >>>>
> > >>>> This is already reflected in the compatibility matrix, if you
> look
> > under SDF.
> > >>>> There is no UnboundedSource interface for portable pipelines.
> That's a
> > legacy
> > >>>> abstraction that will be replaced with SDF.
> > >>>>
> > >>>> Fortunately, there is already a pending PR for cross-language
> > pipelines which
> > >>>> will allow us to use Java IO like PubSub in Python jobs.
> > >>>>
> > >>>> Thanks,
> > >>>> Max
> > >>>>
> > >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> > >>>>> Hey Ankur,
> > >>>>>
> > >>>>> Thanks for the swift reply. Should I change this in the
> capability matrix
> > >>>>> <https://s.apache.org/apache-beam-portability-support-table>
> then?
> > >>>>>
> > >>>>> Many thanks.
> > >>>>> Best,
> > >>>>> Matthias
> > >>>>>
> > >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> > <ma...@google.com>
> > >>>>> <mailto:goenka@google.com <ma...@google.com>>> wrote:
> > >>>>>
> > >>>>> Hi Matthias,
> > >>>>>
> > >>>>> Unfortunately, unbounded reads including pubsub are not
> yet
> > supported for
> > >>>>> portable runners.
> > >>>>>
> > >>>>> Thanks,
> > >>>>> Ankur
> > >>>>>
> > >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens
> > <baetensmatthias@gmail.com <ma...@gmail.com>
> > >>>>> <mailto:baetensmatthias@gmail.com
> > <ma...@gmail.com>>> wrote:
> > >>>>>
> > >>>>> Hi everyone,
> > >>>>>
> > >>>>> Last few days I have been trying to run a streaming
> > pipeline (code on
> > >>>>> Github <https://github.com/matthiasa4/beam-demo>)
> on a
> > Flink Runner.
> > >>>>>
> > >>>>> I am running a Flink cluster locally (v1.5.6
> > >>>>> <https://flink.apache.org/downloads.html>)
> > >>>>> I have built the SDK Harness Container: /./gradlew
> > >>>>> :beam-sdks-python-container:docker/
> > >>>>> and started the JobServer: /./gradlew
> > >>>>> :beam-runners-flink_2.11-job-server:runShadow
> > >>>>> -PflinkMasterUrl=localhost:8081./
> > >>>>>
> > >>>>> I run my pipeline with: /env/bin/python
> streaming_pipeline.py
> > >>>>> --runner=PortableRunner
> --job_endpoint=localhost:8099
> > --output xxx
> > >>>>> --input_subscription xxx --output_subscription xxx/
> > >>>>> /
> > >>>>> /
> > >>>>> All this is running inside a Ubuntu (Bionic) in a
> Virtualbox.
> > >>>>>
> > >>>>> The job submits fine, but unfortunately fails after
> a few
> > seconds with
> > >>>>> the error attached.
> > >>>>>
> > >>>>> Anything I am missing or doing wrong?
> > >>>>>
> > >>>>> Many thanks.
> > >>>>> Best,
> > >>>>> Matthias
> > >>>>>
> > >>>>>
> >
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Wouldn't it be even more useful for the transition period if we enabled Beam IO
to be used via Flink (like in the legacy Flink Runner)? In this particular
example, Matthias wants to use PubSubIO, which is not even available as a native
Flink transform.
On 31.01.19 16:21, Thomas Weise wrote:
> Until SDF is supported, we could also add Flink runner native transforms for
> selected unbounded sources [1].
>
> That might be a reasonable option to unblock users that want to try Python
> streaming on Flink.
>
> Thomas
>
> [1]
> https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
>
>
> On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> > I have a hard time to imagine how can we map in a generic way
> RestrictionTrackers into the existing Bounded/UnboundedSource, so I would
> love to hear more about the details.
>
> Isn't it the other way around? The SDF is a generalization of UnboundedSource.
> So we would wrap UnboundedSource using SDF. I'm not saying it is trivial, but
> SDF offers all the functionality that UnboundedSource needs.
>
> For example, the @GetInitialRestriction method would call split on the
> UnboundedSource and the restriction trackers would then be used to process the
> splits.
>
> On 31.01.19 15:16, Ismaël Mejía wrote:
> >> Not necessarily. This would be one way. Another way is build an SDF
> wrapper for UnboundedSource. Probably the easier path for migration.
> >
> > That would be fantastic, I have heard about such wrapper multiple
> > times but so far there is not any realistic proposal. I have a hard
> > time to imagine how can we map in a generic way RestrictionTrackers
> > into the existing Bounded/UnboundedSource, so I would love to hear
> > more about the details.
> >
> > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
> >>
> >> > In addition to have support in the runners, this will require a
> >> > rewrite of PubsubIO to use the new SDF API.
> >>
> >> Not necessarily. This would be one way. Another way is build an SDF
> wrapper for
> >> UnboundedSource. Probably the easier path for migration.
> >>
> >> On 31.01.19 14:03, Ismaël Mejía wrote:
> >>>> Fortunately, there is already a pending PR for cross-language
> pipelines which
> >>>> will allow us to use Java IO like PubSub in Python jobs.
> >>>
> >>> In addition to have support in the runners, this will require a
> >>> rewrite of PubsubIO to use the new SDF API.
> >>>
> >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
> >>>>
> >>>> Hi Matthias,
> >>>>
> >>>> This is already reflected in the compatibility matrix, if you look
> under SDF.
> >>>> There is no UnboundedSource interface for portable pipelines. That's a
> legacy
> >>>> abstraction that will be replaced with SDF.
> >>>>
> >>>> Fortunately, there is already a pending PR for cross-language
> pipelines which
> >>>> will allow us to use Java IO like PubSub in Python jobs.
> >>>>
> >>>> Thanks,
> >>>> Max
> >>>>
> >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> >>>>> Hey Ankur,
> >>>>>
> >>>>> Thanks for the swift reply. Should I change this in the capability matrix
> >>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
> >>>>>
> >>>>> Many thanks.
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> <ma...@google.com>
> >>>>> <mailto:goenka@google.com <ma...@google.com>>> wrote:
> >>>>>
> >>>>> Hi Matthias,
> >>>>>
> >>>>> Unfortunately, unbounded reads including pubsub are not yet
> supported for
> >>>>> portable runners.
> >>>>>
> >>>>> Thanks,
> >>>>> Ankur
> >>>>>
> >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens
> <baetensmatthias@gmail.com <ma...@gmail.com>
> >>>>> <mailto:baetensmatthias@gmail.com
> <ma...@gmail.com>>> wrote:
> >>>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> Last few days I have been trying to run a streaming
> pipeline (code on
> >>>>> Github <https://github.com/matthiasa4/beam-demo>) on a
> Flink Runner.
> >>>>>
> >>>>> I am running a Flink cluster locally (v1.5.6
> >>>>> <https://flink.apache.org/downloads.html>)
> >>>>> I have built the SDK Harness Container: /./gradlew
> >>>>> :beam-sdks-python-container:docker/
> >>>>> and started the JobServer: /./gradlew
> >>>>> :beam-runners-flink_2.11-job-server:runShadow
> >>>>> -PflinkMasterUrl=localhost:8081./
> >>>>>
> >>>>> I run my pipeline with: /env/bin/python streaming_pipeline.py
> >>>>> --runner=PortableRunner --job_endpoint=localhost:8099
> --output xxx
> >>>>> --input_subscription xxx --output_subscription xxx/
> >>>>> /
> >>>>> /
> >>>>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
> >>>>>
> >>>>> The job submits fine, but unfortunately fails after a few
> seconds with
> >>>>> the error attached.
> >>>>>
> >>>>> Anything I am missing or doing wrong?
> >>>>>
> >>>>> Many thanks.
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Thomas Weise <th...@apache.org>.
Until SDF is supported, we could also add Flink runner native transforms
for selected unbounded sources [1].
That might be a reasonable option to unblock users that want to try Python
streaming on Flink.
Thomas
[1]
https://github.com/lyft/beam/blob/release-2.10.0-lyft/runners/flink/src/main/java/org/apache/beam/runners/flink/LyftFlinkStreamingPortableTranslations.java
On Thu, Jan 31, 2019 at 6:51 AM Maximilian Michels <mx...@apache.org> wrote:
> > I have a hard time to imagine how can we map in a generic way
> RestrictionTrackers into the existing Bounded/UnboundedSource, so I would
> love to hear more about the details.
>
> Isn't it the other way around? The SDF is a generalization of
> UnboundedSource.
> So we would wrap UnboundedSource using SDF. I'm not saying it is trivial,
> but
> SDF offers all the functionality that UnboundedSource needs.
>
> For example, the @GetInitialRestriction method would call split on the
> UnboundedSource and the restriction trackers would then be used to process
> the
> splits.
>
> On 31.01.19 15:16, Ismaël Mejía wrote:
> >> Not necessarily. This would be one way. Another way is build an SDF
> wrapper for UnboundedSource. Probably the easier path for migration.
> >
> > That would be fantastic, I have heard about such wrapper multiple
> > times but so far there is not any realistic proposal. I have a hard
> > time to imagine how can we map in a generic way RestrictionTrackers
> > into the existing Bounded/UnboundedSource, so I would love to hear
> > more about the details.
> >
> > On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>
> >> > In addition to have support in the runners, this will require a
> >> > rewrite of PubsubIO to use the new SDF API.
> >>
> >> Not necessarily. This would be one way. Another way is build an SDF
> wrapper for
> >> UnboundedSource. Probably the easier path for migration.
> >>
> >> On 31.01.19 14:03, Ismaël Mejía wrote:
> >>>> Fortunately, there is already a pending PR for cross-language
> pipelines which
> >>>> will allow us to use Java IO like PubSub in Python jobs.
> >>>
> >>> In addition to have support in the runners, this will require a
> >>> rewrite of PubsubIO to use the new SDF API.
> >>>
> >>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org>
> wrote:
> >>>>
> >>>> Hi Matthias,
> >>>>
> >>>> This is already reflected in the compatibility matrix, if you look
> under SDF.
> >>>> There is no UnboundedSource interface for portable pipelines. That's
> a legacy
> >>>> abstraction that will be replaced with SDF.
> >>>>
> >>>> Fortunately, there is already a pending PR for cross-language
> pipelines which
> >>>> will allow us to use Java IO like PubSub in Python jobs.
> >>>>
> >>>> Thanks,
> >>>> Max
> >>>>
> >>>> On 31.01.19 12:06, Matthias Baetens wrote:
> >>>>> Hey Ankur,
> >>>>>
> >>>>> Thanks for the swift reply. Should I change this in the capability
> matrix
> >>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
> >>>>>
> >>>>> Many thanks.
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> >>>>> <ma...@google.com>> wrote:
> >>>>>
> >>>>> Hi Matthias,
> >>>>>
> >>>>> Unfortunately, unbounded reads including pubsub are not yet
> supported for
> >>>>> portable runners.
> >>>>>
> >>>>> Thanks,
> >>>>> Ankur
> >>>>>
> >>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <
> baetensmatthias@gmail.com
> >>>>> <ma...@gmail.com>> wrote:
> >>>>>
> >>>>> Hi everyone,
> >>>>>
> >>>>> Last few days I have been trying to run a streaming
> pipeline (code on
> >>>>> Github <https://github.com/matthiasa4/beam-demo>) on a
> Flink Runner.
> >>>>>
> >>>>> I am running a Flink cluster locally (v1.5.6
> >>>>> <https://flink.apache.org/downloads.html>)
> >>>>> I have built the SDK Harness Container: /./gradlew
> >>>>> :beam-sdks-python-container:docker/
> >>>>> and started the JobServer: /./gradlew
> >>>>> :beam-runners-flink_2.11-job-server:runShadow
> >>>>> -PflinkMasterUrl=localhost:8081./
> >>>>>
> >>>>> I run my pipeline with: /env/bin/python
> streaming_pipeline.py
> >>>>> --runner=PortableRunner --job_endpoint=localhost:8099
> --output xxx
> >>>>> --input_subscription xxx --output_subscription xxx/
> >>>>> /
> >>>>> /
> >>>>> All this is running inside a Ubuntu (Bionic) in a
> Virtualbox.
> >>>>>
> >>>>> The job submits fine, but unfortunately fails after a few
> seconds with
> >>>>> the error attached.
> >>>>>
> >>>>> Anything I am missing or doing wrong?
> >>>>>
> >>>>> Many thanks.
> >>>>> Best,
> >>>>> Matthias
> >>>>>
> >>>>>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
> I have a hard time to imagine how can we map in a generic way RestrictionTrackers into the existing Bounded/UnboundedSource, so I would love to hear more about the details.
Isn't it the other way around? The SDF is a generalization of UnboundedSource.
So we would wrap UnboundedSource using SDF. I'm not saying it is trivial, but
SDF offers all the functionality that UnboundedSource needs.
For example, the @GetInitialRestriction method would call split on the
UnboundedSource and the restriction trackers would then be used to process the
splits.
On 31.01.19 15:16, Ismaël Mejía wrote:
>> Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration.
>
> That would be fantastic, I have heard about such wrapper multiple
> times but so far there is not any realistic proposal. I have a hard
> time to imagine how can we map in a generic way RestrictionTrackers
> into the existing Bounded/UnboundedSource, so I would love to hear
> more about the details.
>
> On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> > In addition to have support in the runners, this will require a
>> > rewrite of PubsubIO to use the new SDF API.
>>
>> Not necessarily. This would be one way. Another way is build an SDF wrapper for
>> UnboundedSource. Probably the easier path for migration.
>>
>> On 31.01.19 14:03, Ismaël Mejía wrote:
>>>> Fortunately, there is already a pending PR for cross-language pipelines which
>>>> will allow us to use Java IO like PubSub in Python jobs.
>>>
>>> In addition to have support in the runners, this will require a
>>> rewrite of PubsubIO to use the new SDF API.
>>>
>>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
>>>>
>>>> Hi Matthias,
>>>>
>>>> This is already reflected in the compatibility matrix, if you look under SDF.
>>>> There is no UnboundedSource interface for portable pipelines. That's a legacy
>>>> abstraction that will be replaced with SDF.
>>>>
>>>> Fortunately, there is already a pending PR for cross-language pipelines which
>>>> will allow us to use Java IO like PubSub in Python jobs.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>> On 31.01.19 12:06, Matthias Baetens wrote:
>>>>> Hey Ankur,
>>>>>
>>>>> Thanks for the swift reply. Should I change this in the capability matrix
>>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
>>>>>
>>>>> Many thanks.
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
>>>>> <ma...@google.com>> wrote:
>>>>>
>>>>> Hi Matthias,
>>>>>
>>>>> Unfortunately, unbounded reads including pubsub are not yet supported for
>>>>> portable runners.
>>>>>
>>>>> Thanks,
>>>>> Ankur
>>>>>
>>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Last few days I have been trying to run a streaming pipeline (code on
>>>>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>>>>
>>>>> I am running a Flink cluster locally (v1.5.6
>>>>> <https://flink.apache.org/downloads.html>)
>>>>> I have built the SDK Harness Container: /./gradlew
>>>>> :beam-sdks-python-container:docker/
>>>>> and started the JobServer: /./gradlew
>>>>> :beam-runners-flink_2.11-job-server:runShadow
>>>>> -PflinkMasterUrl=localhost:8081./
>>>>>
>>>>> I run my pipeline with: /env/bin/python streaming_pipeline.py
>>>>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>>>>> --input_subscription xxx --output_subscription xxx/
>>>>> /
>>>>> /
>>>>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>>>>
>>>>> The job submits fine, but unfortunately fails after a few seconds with
>>>>> the error attached.
>>>>>
>>>>> Anything I am missing or doing wrong?
>>>>>
>>>>> Many thanks.
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
> I have a hard time to imagine how can we map in a generic way RestrictionTrackers into the existing Bounded/UnboundedSource, so I would love to hear more about the details.
Isn't it the other way around? The SDF is a generalization of UnboundedSource.
So we would wrap UnboundedSource using SDF. I'm not saying it is trivial, but
SDF offers all the functionality that UnboundedSource needs.
For example, the @GetInitialRestriction method would call split on the
UnboundedSource and the restriction trackers would then be used to process the
splits.
On 31.01.19 15:16, Ismaël Mejía wrote:
>> Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration.
>
> That would be fantastic, I have heard about such wrapper multiple
> times but so far there is not any realistic proposal. I have a hard
> time to imagine how can we map in a generic way RestrictionTrackers
> into the existing Bounded/UnboundedSource, so I would love to hear
> more about the details.
>
> On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> > In addition to have support in the runners, this will require a
>> > rewrite of PubsubIO to use the new SDF API.
>>
>> Not necessarily. This would be one way. Another way is build an SDF wrapper for
>> UnboundedSource. Probably the easier path for migration.
>>
>> On 31.01.19 14:03, Ismaël Mejía wrote:
>>>> Fortunately, there is already a pending PR for cross-language pipelines which
>>>> will allow us to use Java IO like PubSub in Python jobs.
>>>
>>> In addition to have support in the runners, this will require a
>>> rewrite of PubsubIO to use the new SDF API.
>>>
>>> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
>>>>
>>>> Hi Matthias,
>>>>
>>>> This is already reflected in the compatibility matrix, if you look under SDF.
>>>> There is no UnboundedSource interface for portable pipelines. That's a legacy
>>>> abstraction that will be replaced with SDF.
>>>>
>>>> Fortunately, there is already a pending PR for cross-language pipelines which
>>>> will allow us to use Java IO like PubSub in Python jobs.
>>>>
>>>> Thanks,
>>>> Max
>>>>
>>>> On 31.01.19 12:06, Matthias Baetens wrote:
>>>>> Hey Ankur,
>>>>>
>>>>> Thanks for the swift reply. Should I change this in the capability matrix
>>>>> <https://s.apache.org/apache-beam-portability-support-table> then?
>>>>>
>>>>> Many thanks.
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
>>>>> <ma...@google.com>> wrote:
>>>>>
>>>>> Hi Matthias,
>>>>>
>>>>> Unfortunately, unbounded reads including pubsub are not yet supported for
>>>>> portable runners.
>>>>>
>>>>> Thanks,
>>>>> Ankur
>>>>>
>>>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
>>>>> <ma...@gmail.com>> wrote:
>>>>>
>>>>> Hi everyone,
>>>>>
>>>>> Last few days I have been trying to run a streaming pipeline (code on
>>>>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>>>>
>>>>> I am running a Flink cluster locally (v1.5.6
>>>>> <https://flink.apache.org/downloads.html>)
>>>>> I have built the SDK Harness Container: /./gradlew
>>>>> :beam-sdks-python-container:docker/
>>>>> and started the JobServer: /./gradlew
>>>>> :beam-runners-flink_2.11-job-server:runShadow
>>>>> -PflinkMasterUrl=localhost:8081./
>>>>>
>>>>> I run my pipeline with: /env/bin/python streaming_pipeline.py
>>>>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>>>>> --input_subscription xxx --output_subscription xxx/
>>>>> /
>>>>> /
>>>>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>>>>
>>>>> The job submits fine, but unfortunately fails after a few seconds with
>>>>> the error attached.
>>>>>
>>>>> Anything I am missing or doing wrong?
>>>>>
>>>>> Many thanks.
>>>>> Best,
>>>>> Matthias
>>>>>
>>>>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
> Not necessarily. This would be one way. Another way is build an SDF wrapper for UnboundedSource. Probably the easier path for migration.
That would be fantastic, I have heard about such wrapper multiple
times but so far there is not any realistic proposal. I have a hard
time to imagine how can we map in a generic way RestrictionTrackers
into the existing Bounded/UnboundedSource, so I would love to hear
more about the details.
On Thu, Jan 31, 2019 at 3:07 PM Maximilian Michels <mx...@apache.org> wrote:
>
> > In addition to have support in the runners, this will require a
> > rewrite of PubsubIO to use the new SDF API.
>
> Not necessarily. This would be one way. Another way is build an SDF wrapper for
> UnboundedSource. Probably the easier path for migration.
>
> On 31.01.19 14:03, Ismaël Mejía wrote:
> >> Fortunately, there is already a pending PR for cross-language pipelines which
> >> will allow us to use Java IO like PubSub in Python jobs.
> >
> > In addition to have support in the runners, this will require a
> > rewrite of PubsubIO to use the new SDF API.
> >
> > On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
> >>
> >> Hi Matthias,
> >>
> >> This is already reflected in the compatibility matrix, if you look under SDF.
> >> There is no UnboundedSource interface for portable pipelines. That's a legacy
> >> abstraction that will be replaced with SDF.
> >>
> >> Fortunately, there is already a pending PR for cross-language pipelines which
> >> will allow us to use Java IO like PubSub in Python jobs.
> >>
> >> Thanks,
> >> Max
> >>
> >> On 31.01.19 12:06, Matthias Baetens wrote:
> >>> Hey Ankur,
> >>>
> >>> Thanks for the swift reply. Should I change this in the capability matrix
> >>> <https://s.apache.org/apache-beam-portability-support-table> then?
> >>>
> >>> Many thanks.
> >>> Best,
> >>> Matthias
> >>>
> >>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> >>> <ma...@google.com>> wrote:
> >>>
> >>> Hi Matthias,
> >>>
> >>> Unfortunately, unbounded reads including pubsub are not yet supported for
> >>> portable runners.
> >>>
> >>> Thanks,
> >>> Ankur
> >>>
> >>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> >>> <ma...@gmail.com>> wrote:
> >>>
> >>> Hi everyone,
> >>>
> >>> Last few days I have been trying to run a streaming pipeline (code on
> >>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
> >>>
> >>> I am running a Flink cluster locally (v1.5.6
> >>> <https://flink.apache.org/downloads.html>)
> >>> I have built the SDK Harness Container: /./gradlew
> >>> :beam-sdks-python-container:docker/
> >>> and started the JobServer: /./gradlew
> >>> :beam-runners-flink_2.11-job-server:runShadow
> >>> -PflinkMasterUrl=localhost:8081./
> >>>
> >>> I run my pipeline with: /env/bin/python streaming_pipeline.py
> >>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> >>> --input_subscription xxx --output_subscription xxx/
> >>> /
> >>> /
> >>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
> >>>
> >>> The job submits fine, but unfortunately fails after a few seconds with
> >>> the error attached.
> >>>
> >>> Anything I am missing or doing wrong?
> >>>
> >>> Many thanks.
> >>> Best,
> >>> Matthias
> >>>
> >>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
> In addition to have support in the runners, this will require a
> rewrite of PubsubIO to use the new SDF API.
Not necessarily. This would be one way. Another way is build an SDF wrapper for
UnboundedSource. Probably the easier path for migration.
On 31.01.19 14:03, Ismaël Mejía wrote:
>> Fortunately, there is already a pending PR for cross-language pipelines which
>> will allow us to use Java IO like PubSub in Python jobs.
>
> In addition to have support in the runners, this will require a
> rewrite of PubsubIO to use the new SDF API.
>
> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi Matthias,
>>
>> This is already reflected in the compatibility matrix, if you look under SDF.
>> There is no UnboundedSource interface for portable pipelines. That's a legacy
>> abstraction that will be replaced with SDF.
>>
>> Fortunately, there is already a pending PR for cross-language pipelines which
>> will allow us to use Java IO like PubSub in Python jobs.
>>
>> Thanks,
>> Max
>>
>> On 31.01.19 12:06, Matthias Baetens wrote:
>>> Hey Ankur,
>>>
>>> Thanks for the swift reply. Should I change this in the capability matrix
>>> <https://s.apache.org/apache-beam-portability-support-table> then?
>>>
>>> Many thanks.
>>> Best,
>>> Matthias
>>>
>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
>>> <ma...@google.com>> wrote:
>>>
>>> Hi Matthias,
>>>
>>> Unfortunately, unbounded reads including pubsub are not yet supported for
>>> portable runners.
>>>
>>> Thanks,
>>> Ankur
>>>
>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hi everyone,
>>>
>>> Last few days I have been trying to run a streaming pipeline (code on
>>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>>
>>> I am running a Flink cluster locally (v1.5.6
>>> <https://flink.apache.org/downloads.html>)
>>> I have built the SDK Harness Container: /./gradlew
>>> :beam-sdks-python-container:docker/
>>> and started the JobServer: /./gradlew
>>> :beam-runners-flink_2.11-job-server:runShadow
>>> -PflinkMasterUrl=localhost:8081./
>>>
>>> I run my pipeline with: /env/bin/python streaming_pipeline.py
>>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>>> --input_subscription xxx --output_subscription xxx/
>>> /
>>> /
>>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>>
>>> The job submits fine, but unfortunately fails after a few seconds with
>>> the error attached.
>>>
>>> Anything I am missing or doing wrong?
>>>
>>> Many thanks.
>>> Best,
>>> Matthias
>>>
>>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
> In addition to have support in the runners, this will require a
> rewrite of PubsubIO to use the new SDF API.
Not necessarily. This would be one way. Another way is build an SDF wrapper for
UnboundedSource. Probably the easier path for migration.
On 31.01.19 14:03, Ismaël Mejía wrote:
>> Fortunately, there is already a pending PR for cross-language pipelines which
>> will allow us to use Java IO like PubSub in Python jobs.
>
> In addition to have support in the runners, this will require a
> rewrite of PubsubIO to use the new SDF API.
>
> On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>> Hi Matthias,
>>
>> This is already reflected in the compatibility matrix, if you look under SDF.
>> There is no UnboundedSource interface for portable pipelines. That's a legacy
>> abstraction that will be replaced with SDF.
>>
>> Fortunately, there is already a pending PR for cross-language pipelines which
>> will allow us to use Java IO like PubSub in Python jobs.
>>
>> Thanks,
>> Max
>>
>> On 31.01.19 12:06, Matthias Baetens wrote:
>>> Hey Ankur,
>>>
>>> Thanks for the swift reply. Should I change this in the capability matrix
>>> <https://s.apache.org/apache-beam-portability-support-table> then?
>>>
>>> Many thanks.
>>> Best,
>>> Matthias
>>>
>>> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
>>> <ma...@google.com>> wrote:
>>>
>>> Hi Matthias,
>>>
>>> Unfortunately, unbounded reads including pubsub are not yet supported for
>>> portable runners.
>>>
>>> Thanks,
>>> Ankur
>>>
>>> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
>>> <ma...@gmail.com>> wrote:
>>>
>>> Hi everyone,
>>>
>>> Last few days I have been trying to run a streaming pipeline (code on
>>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>>
>>> I am running a Flink cluster locally (v1.5.6
>>> <https://flink.apache.org/downloads.html>)
>>> I have built the SDK Harness Container: /./gradlew
>>> :beam-sdks-python-container:docker/
>>> and started the JobServer: /./gradlew
>>> :beam-runners-flink_2.11-job-server:runShadow
>>> -PflinkMasterUrl=localhost:8081./
>>>
>>> I run my pipeline with: /env/bin/python streaming_pipeline.py
>>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>>> --input_subscription xxx --output_subscription xxx/
>>> /
>>> /
>>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>>
>>> The job submits fine, but unfortunately fails after a few seconds with
>>> the error attached.
>>>
>>> Anything I am missing or doing wrong?
>>>
>>> Many thanks.
>>> Best,
>>> Matthias
>>>
>>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ismaël Mejía <ie...@gmail.com>.
> Fortunately, there is already a pending PR for cross-language pipelines which
> will allow us to use Java IO like PubSub in Python jobs.
In addition to have support in the runners, this will require a
rewrite of PubsubIO to use the new SDF API.
On Thu, Jan 31, 2019 at 12:23 PM Maximilian Michels <mx...@apache.org> wrote:
>
> Hi Matthias,
>
> This is already reflected in the compatibility matrix, if you look under SDF.
> There is no UnboundedSource interface for portable pipelines. That's a legacy
> abstraction that will be replaced with SDF.
>
> Fortunately, there is already a pending PR for cross-language pipelines which
> will allow us to use Java IO like PubSub in Python jobs.
>
> Thanks,
> Max
>
> On 31.01.19 12:06, Matthias Baetens wrote:
> > Hey Ankur,
> >
> > Thanks for the swift reply. Should I change this in the capability matrix
> > <https://s.apache.org/apache-beam-portability-support-table> then?
> >
> > Many thanks.
> > Best,
> > Matthias
> >
> > On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> > <ma...@google.com>> wrote:
> >
> > Hi Matthias,
> >
> > Unfortunately, unbounded reads including pubsub are not yet supported for
> > portable runners.
> >
> > Thanks,
> > Ankur
> >
> > On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> > <ma...@gmail.com>> wrote:
> >
> > Hi everyone,
> >
> > Last few days I have been trying to run a streaming pipeline (code on
> > Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
> >
> > I am running a Flink cluster locally (v1.5.6
> > <https://flink.apache.org/downloads.html>)
> > I have built the SDK Harness Container: /./gradlew
> > :beam-sdks-python-container:docker/
> > and started the JobServer: /./gradlew
> > :beam-runners-flink_2.11-job-server:runShadow
> > -PflinkMasterUrl=localhost:8081./
> >
> > I run my pipeline with: /env/bin/python streaming_pipeline.py
> > --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> > --input_subscription xxx --output_subscription xxx/
> > /
> > /
> > All this is running inside a Ubuntu (Bionic) in a Virtualbox.
> >
> > The job submits fine, but unfortunately fails after a few seconds with
> > the error attached.
> >
> > Anything I am missing or doing wrong?
> >
> > Many thanks.
> > Best,
> > Matthias
> >
> >
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Hi Matthias,
This is already reflected in the compatibility matrix, if you look under SDF.
There is no UnboundedSource interface for portable pipelines. That's a legacy
abstraction that will be replaced with SDF.
Fortunately, there is already a pending PR for cross-language pipelines which
will allow us to use Java IO like PubSub in Python jobs.
Thanks,
Max
On 31.01.19 12:06, Matthias Baetens wrote:
> Hey Ankur,
>
> Thanks for the swift reply. Should I change this in the capability matrix
> <https://s.apache.org/apache-beam-portability-support-table> then?
>
> Many thanks.
> Best,
> Matthias
>
> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> <ma...@google.com>> wrote:
>
> Hi Matthias,
>
> Unfortunately, unbounded reads including pubsub are not yet supported for
> portable runners.
>
> Thanks,
> Ankur
>
> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi everyone,
>
> Last few days I have been trying to run a streaming pipeline (code on
> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>
> I am running a Flink cluster locally (v1.5.6
> <https://flink.apache.org/downloads.html>)
> I have built the SDK Harness Container: /./gradlew
> :beam-sdks-python-container:docker/
> and started the JobServer: /./gradlew
> :beam-runners-flink_2.11-job-server:runShadow
> -PflinkMasterUrl=localhost:8081./
>
> I run my pipeline with: /env/bin/python streaming_pipeline.py
> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> --input_subscription xxx --output_subscription xxx/
> /
> /
> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>
> The job submits fine, but unfortunately fails after a few seconds with
> the error attached.
>
> Anything I am missing or doing wrong?
>
> Many thanks.
> Best,
> Matthias
>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Maximilian Michels <mx...@apache.org>.
Hi Matthias,
This is already reflected in the compatibility matrix, if you look under SDF.
There is no UnboundedSource interface for portable pipelines. That's a legacy
abstraction that will be replaced with SDF.
Fortunately, there is already a pending PR for cross-language pipelines which
will allow us to use Java IO like PubSub in Python jobs.
Thanks,
Max
On 31.01.19 12:06, Matthias Baetens wrote:
> Hey Ankur,
>
> Thanks for the swift reply. Should I change this in the capability matrix
> <https://s.apache.org/apache-beam-portability-support-table> then?
>
> Many thanks.
> Best,
> Matthias
>
> On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <goenka@google.com
> <ma...@google.com>> wrote:
>
> Hi Matthias,
>
> Unfortunately, unbounded reads including pubsub are not yet supported for
> portable runners.
>
> Thanks,
> Ankur
>
> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <baetensmatthias@gmail.com
> <ma...@gmail.com>> wrote:
>
> Hi everyone,
>
> Last few days I have been trying to run a streaming pipeline (code on
> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>
> I am running a Flink cluster locally (v1.5.6
> <https://flink.apache.org/downloads.html>)
> I have built the SDK Harness Container: /./gradlew
> :beam-sdks-python-container:docker/
> and started the JobServer: /./gradlew
> :beam-runners-flink_2.11-job-server:runShadow
> -PflinkMasterUrl=localhost:8081./
>
> I run my pipeline with: /env/bin/python streaming_pipeline.py
> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> --input_subscription xxx --output_subscription xxx/
> /
> /
> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>
> The job submits fine, but unfortunately fails after a few seconds with
> the error attached.
>
> Anything I am missing or doing wrong?
>
> Many thanks.
> Best,
> Matthias
>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Matthias Baetens <ba...@gmail.com>.
Hey Ankur,
Thanks for the swift reply. Should I change this in the capability matrix
<https://s.apache.org/apache-beam-portability-support-table> then?
Many thanks.
Best,
Matthias
On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <go...@google.com> wrote:
> Hi Matthias,
>
> Unfortunately, unbounded reads including pubsub are not yet supported for
> portable runners.
>
> Thanks,
> Ankur
>
> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <
> baetensmatthias@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Last few days I have been trying to run a streaming pipeline (code on
>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>
>> I am running a Flink cluster locally (v1.5.6
>> <https://flink.apache.org/downloads.html>)
>> I have built the SDK Harness Container: *./gradlew
>> :beam-sdks-python-container:docker*
>> and started the JobServer: *./gradlew
>> :beam-runners-flink_2.11-job-server:runShadow
>> -PflinkMasterUrl=localhost:8081.*
>>
>> I run my pipeline with: *env/bin/python streaming_pipeline.py
>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>> --input_subscription xxx --output_subscription xxx*
>>
>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>
>> The job submits fine, but unfortunately fails after a few seconds with
>> the error attached.
>>
>> Anything I am missing or doing wrong?
>>
>> Many thanks.
>> Best,
>> Matthias
>>
>>
>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Matthias Baetens <ba...@gmail.com>.
Hey Ankur,
Thanks for the swift reply. Should I change this in the capability matrix
<https://s.apache.org/apache-beam-portability-support-table> then?
Many thanks.
Best,
Matthias
On Thu, 31 Jan 2019 at 09:31, Ankur Goenka <go...@google.com> wrote:
> Hi Matthias,
>
> Unfortunately, unbounded reads including pubsub are not yet supported for
> portable runners.
>
> Thanks,
> Ankur
>
> On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <
> baetensmatthias@gmail.com> wrote:
>
>> Hi everyone,
>>
>> Last few days I have been trying to run a streaming pipeline (code on
>> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>>
>> I am running a Flink cluster locally (v1.5.6
>> <https://flink.apache.org/downloads.html>)
>> I have built the SDK Harness Container: *./gradlew
>> :beam-sdks-python-container:docker*
>> and started the JobServer: *./gradlew
>> :beam-runners-flink_2.11-job-server:runShadow
>> -PflinkMasterUrl=localhost:8081.*
>>
>> I run my pipeline with: *env/bin/python streaming_pipeline.py
>> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
>> --input_subscription xxx --output_subscription xxx*
>>
>> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>>
>> The job submits fine, but unfortunately fails after a few seconds with
>> the error attached.
>>
>> Anything I am missing or doing wrong?
>>
>> Many thanks.
>> Best,
>> Matthias
>>
>>
>>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ankur Goenka <go...@google.com>.
Hi Matthias,
Unfortunately, unbounded reads including pubsub are not yet supported for
portable runners.
Thanks,
Ankur
On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <ba...@gmail.com>
wrote:
> Hi everyone,
>
> Last few days I have been trying to run a streaming pipeline (code on
> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>
> I am running a Flink cluster locally (v1.5.6
> <https://flink.apache.org/downloads.html>)
> I have built the SDK Harness Container: *./gradlew
> :beam-sdks-python-container:docker*
> and started the JobServer: *./gradlew
> :beam-runners-flink_2.11-job-server:runShadow
> -PflinkMasterUrl=localhost:8081.*
>
> I run my pipeline with: *env/bin/python streaming_pipeline.py
> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> --input_subscription xxx --output_subscription xxx*
>
> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>
> The job submits fine, but unfortunately fails after a few seconds with the
> error attached.
>
> Anything I am missing or doing wrong?
>
> Many thanks.
> Best,
> Matthias
>
>
>
Re: Beam Python streaming pipeline on Flink Runner
Posted by Ankur Goenka <go...@google.com>.
Hi Matthias,
Unfortunately, unbounded reads including pubsub are not yet supported for
portable runners.
Thanks,
Ankur
On Thu, Jan 31, 2019 at 2:44 PM Matthias Baetens <ba...@gmail.com>
wrote:
> Hi everyone,
>
> Last few days I have been trying to run a streaming pipeline (code on
> Github <https://github.com/matthiasa4/beam-demo>) on a Flink Runner.
>
> I am running a Flink cluster locally (v1.5.6
> <https://flink.apache.org/downloads.html>)
> I have built the SDK Harness Container: *./gradlew
> :beam-sdks-python-container:docker*
> and started the JobServer: *./gradlew
> :beam-runners-flink_2.11-job-server:runShadow
> -PflinkMasterUrl=localhost:8081.*
>
> I run my pipeline with: *env/bin/python streaming_pipeline.py
> --runner=PortableRunner --job_endpoint=localhost:8099 --output xxx
> --input_subscription xxx --output_subscription xxx*
>
> All this is running inside a Ubuntu (Bionic) in a Virtualbox.
>
> The job submits fine, but unfortunately fails after a few seconds with the
> error attached.
>
> Anything I am missing or doing wrong?
>
> Many thanks.
> Best,
> Matthias
>
>
>