You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Maximilian Michels <mx...@apache.org> on 2019/05/22 16:17:32 UTC
Environments for External Transforms
Hi,
Robert and me were discussing on the subject of user-specified
environments for external transforms [1]. We couldn't decide whether
users should have direct control over the environment when they use an
external transform in their pipeline.
In my mind, it is quite natural that the Expansion Service is a
long-running service that gets started with a list of available
environments. Such a list can be outdated and users may write transforms
for a new environment they want to use in their pipeline. The easiest
way would be to allow to pass the environment with the transform. Note
that we already give users control over the "main" environment via the
PortablePipelineOptions, so this wouldn't be an entirely new concept.
The contrary position is that the Expansion Service should have full
control over which environment is chosen. Going back to the discussion
about artifact staging [2], this could enable to perform more
optimizations, such as merging environments or detecting conflicts.
However, this only works if this information has been provided upfront
to the Expansion Service. It wouldn't be impossible to provide these
hints alongside with the environment like suggested in the previous
paragraph.
Any opinions? Should we allow users to optionally specify an environment
for external transforms?
Thanks,
Max
[1] https://github.com/apache/beam/pull/8639
[2]
https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
Re: Environments for External Transforms
Posted by Robert Bradshaw <ro...@google.com>.
On Mon, May 27, 2019 at 1:59 PM Maximilian Michels <mx...@apache.org> wrote:
>
> > I think it makes a lot of sense for job servers to also act as
> > expansion services, but one can't of course defer expansion until job
> > submission.
>
> One could defer the expansion until job submission but it would be a
> semantic change to how expansion works currently. In particular with
> respect to providing feedback to the user during expansion and with
> regard to immutability of pipelines, this would not be a good choice.
It's also not always possible, e.g. the results (their types in
particular, but also even how many) may not be known until after
expansion.
> As for the Flink job server, it already hosts an expansion server. It
> would make sense to let them share the same GRPC server which would
> avoid having to know the port of the expansion server.
+1
And, on a pragmatic note, it'd be good to share the port with the
artifact server as well, in which case the job server could say "serve
artifacts to me" without having to worry about any intervening port
forwarding, etc. that sits between it and the sdk.
> On 27.05.19 13:33, Robert Bradshaw wrote:
> > On Mon, May 27, 2019 at 12:38 PM Maximilian Michels <mx...@apache.org> wrote:
> >>
> >>> Which environment would be used to perform the expansion? I think this is an interesting option, as long as it does not introduce a hard dependency on docker.
> >>
> >> The same environment that the to-be-expanded transform requires during
> >> runtime.
> >>
> >>> Dataflow has been doing something similar in this route where it is trying to get rid of the driver program running on the users machine. If you can get the expansion service to launch and run an environment to perform the expansion, you could also get it to create and submit a job as well returning data around the running job.
> >>
> >> Portability already runs without a driver on the user machine, apart
> >> from expansion and staging. For anything runtime-related the job server
> >> kicks in. It's worth to think about delegating expansion and staging to
> >> the Job server.
> >
> > I think it makes a lot of sense for job servers to also act as
> > expansion services, but one can't of course defer expansion until job
> > submission.
> >
> >> On 24.05.19 23:48, Lukasz Cwik wrote:
> >>> Dataflow has been doing something similar in this route where it is
> >>> trying to get rid of the driver program running on the users machine. If
> >>> you can get the expansion service to launch and run an environment to
> >>> perform the expansion, you could also get it to create and submit a job
> >>> as well returning data around the running job.
> >>>
> >>> On Thu, May 23, 2019 at 7:47 AM Thomas Weise <thw@apache.org
> >>> <ma...@apache.org>> wrote:
> >>>
> >>>
> >>>
> >>> On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mxm@apache.org
> >>> <ma...@apache.org>> wrote:
> >>>
> >>> > Writing a new transform involves updating the expansion
> >>> service to include their new transform.
> >>>
> >>> Would it be conceivable that the expansion is performed via the
> >>> environment? That would solve the problem of updating the expansion
> >>> service, although it adds additional complexity for bringing up the
> >>> environment.
> >>>
> >>>
> >>> Which environment would be used to perform the expansion? I think
> >>> this is an interesting option, as long as it does not introduce a
> >>> hard dependency on docker.
> >>>
> >>> On 23.05.19 11:31, Robert Bradshaw wrote:
> >>> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels
> >>> <mxm@apache.org <ma...@apache.org>
> >>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >>> >
> >>> > Hi,
> >>> >
> >>> > Robert and me were discussing on the subject of
> >>> user-specified
> >>> > environments for external transforms [1]. We couldn't
> >>> decide whether
> >>> > users should have direct control over the environment
> >>> when they use an
> >>> > external transform in their pipeline.
> >>> >
> >>> > In my mind, it is quite natural that the Expansion
> >>> Service is a
> >>> > long-running service that gets started with a list of
> >>> available
> >>> > environments.
> >>> >
> >>> >
> >>> > +1.
> >>> >
> >>> > IMHO, the expansion service should be expected to provide valid
> >>> > environments for the transforms it vendors. Removing this
> >>> expectation
> >>> > seems wrong. Making it cheap to specify non-default
> >>> dependencies without
> >>> > building (publishing, etc.) a docker image is probably key to
> >>> making
> >>> > this work well (and also allowing more powerful environment
> >>> introspection).
> >>> >
> >>> > Such a list can be outdated and users may write transforms
> >>> > for a new environment they want to use in their pipeline.
> >>> >
> >>> >
> >>> > This is the part that I'm having trouble following. Writing a
> >>> new
> >>> > transform involves updating the expansion service to include
> >>> their new
> >>> > transform. The author of a transform (in other words, the one
> >>> who
> >>> > defines its expansion and implementation) is in the position
> >>> to name its
> >>> > dependencies, etc. and the user of the transform (the one
> >>> invoking it)
> >>> > is not in a generally good position to know what environments
> >>> would be
> >>> > valid.
> >>> >
> >>> > The easiest
> >>> > way would be to allow to pass the environment with the
> >>> transform.
> >>> >
> >>> >
> >>> > What this allows is using existing transforms in new
> >>> environments. There
> >>> > are possibly some usecases for this, e.g. expansion of a
> >>> given transform
> >>> > may be compatible with ether version X or version Y of a
> >>> library, left
> >>> > up to the discretion of the caller, but I think that this is
> >>> really just
> >>> > a deficiency in our environment specifications (e.g. it one
> >>> should be
> >>> > able to express this flexibility in the returned environment).
> >>> >
> >>> > Note
> >>> > that we already give users control over the "main"
> >>> environment via the
> >>> > PortablePipelineOptions, so this wouldn't be an entirely
> >>> new concept.
> >>> >
> >>> >
> >>> > Yes, the author of a pipeline/transform chooses the
> >>> environment in which
> >>> > those transforms execute.
> >>> >
> >>> > The contrary position is that the Expansion Service
> >>> should have full
> >>> > control over which environment is chosen. Going back to
> >>> the discussion
> >>> > about artifact staging [2], this could enable to perform more
> >>> > optimizations, such as merging environments or detecting
> >>> conflicts.
> >>> > However, this only works if this information has been
> >>> provided upfront
> >>> > to the Expansion Service. It wouldn't be impossible to
> >>> provide these
> >>> > hints alongside with the environment like suggested in
> >>> the previous
> >>> > paragraph.
> >>> >
> >>> > Any opinions? Should we allow users to optionally specify an
> >>> > environment
> >>> > for external transforms?
> >>> >
> >>> > Thanks,
> >>> > Max
> >>> >
> >>> > [1] https://github.com/apache/beam/pull/8639
> >>> > [2]
> >>> >
> >>> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> >>> >
> >>>
Re: Environments for External Transforms
Posted by Maximilian Michels <mx...@apache.org>.
> I think it makes a lot of sense for job servers to also act as
> expansion services, but one can't of course defer expansion until job
> submission.
One could defer the expansion until job submission but it would be a
semantic change to how expansion works currently. In particular with
respect to providing feedback to the user during expansion and with
regard to immutability of pipelines, this would not be a good choice.
As for the Flink job server, it already hosts an expansion server. It
would make sense to let them share the same GRPC server which would
avoid having to know the port of the expansion server.
On 27.05.19 13:33, Robert Bradshaw wrote:
> On Mon, May 27, 2019 at 12:38 PM Maximilian Michels <mx...@apache.org> wrote:
>>
>>> Which environment would be used to perform the expansion? I think this is an interesting option, as long as it does not introduce a hard dependency on docker.
>>
>> The same environment that the to-be-expanded transform requires during
>> runtime.
>>
>>> Dataflow has been doing something similar in this route where it is trying to get rid of the driver program running on the users machine. If you can get the expansion service to launch and run an environment to perform the expansion, you could also get it to create and submit a job as well returning data around the running job.
>>
>> Portability already runs without a driver on the user machine, apart
>> from expansion and staging. For anything runtime-related the job server
>> kicks in. It's worth to think about delegating expansion and staging to
>> the Job server.
>
> I think it makes a lot of sense for job servers to also act as
> expansion services, but one can't of course defer expansion until job
> submission.
>
>> On 24.05.19 23:48, Lukasz Cwik wrote:
>>> Dataflow has been doing something similar in this route where it is
>>> trying to get rid of the driver program running on the users machine. If
>>> you can get the expansion service to launch and run an environment to
>>> perform the expansion, you could also get it to create and submit a job
>>> as well returning data around the running job.
>>>
>>> On Thu, May 23, 2019 at 7:47 AM Thomas Weise <thw@apache.org
>>> <ma...@apache.org>> wrote:
>>>
>>>
>>>
>>> On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mxm@apache.org
>>> <ma...@apache.org>> wrote:
>>>
>>> > Writing a new transform involves updating the expansion
>>> service to include their new transform.
>>>
>>> Would it be conceivable that the expansion is performed via the
>>> environment? That would solve the problem of updating the expansion
>>> service, although it adds additional complexity for bringing up the
>>> environment.
>>>
>>>
>>> Which environment would be used to perform the expansion? I think
>>> this is an interesting option, as long as it does not introduce a
>>> hard dependency on docker.
>>>
>>> On 23.05.19 11:31, Robert Bradshaw wrote:
>>> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels
>>> <mxm@apache.org <ma...@apache.org>
>>> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
>>> >
>>> > Hi,
>>> >
>>> > Robert and me were discussing on the subject of
>>> user-specified
>>> > environments for external transforms [1]. We couldn't
>>> decide whether
>>> > users should have direct control over the environment
>>> when they use an
>>> > external transform in their pipeline.
>>> >
>>> > In my mind, it is quite natural that the Expansion
>>> Service is a
>>> > long-running service that gets started with a list of
>>> available
>>> > environments.
>>> >
>>> >
>>> > +1.
>>> >
>>> > IMHO, the expansion service should be expected to provide valid
>>> > environments for the transforms it vendors. Removing this
>>> expectation
>>> > seems wrong. Making it cheap to specify non-default
>>> dependencies without
>>> > building (publishing, etc.) a docker image is probably key to
>>> making
>>> > this work well (and also allowing more powerful environment
>>> introspection).
>>> >
>>> > Such a list can be outdated and users may write transforms
>>> > for a new environment they want to use in their pipeline.
>>> >
>>> >
>>> > This is the part that I'm having trouble following. Writing a
>>> new
>>> > transform involves updating the expansion service to include
>>> their new
>>> > transform. The author of a transform (in other words, the one
>>> who
>>> > defines its expansion and implementation) is in the position
>>> to name its
>>> > dependencies, etc. and the user of the transform (the one
>>> invoking it)
>>> > is not in a generally good position to know what environments
>>> would be
>>> > valid.
>>> >
>>> > The easiest
>>> > way would be to allow to pass the environment with the
>>> transform.
>>> >
>>> >
>>> > What this allows is using existing transforms in new
>>> environments. There
>>> > are possibly some usecases for this, e.g. expansion of a
>>> given transform
>>> > may be compatible with ether version X or version Y of a
>>> library, left
>>> > up to the discretion of the caller, but I think that this is
>>> really just
>>> > a deficiency in our environment specifications (e.g. it one
>>> should be
>>> > able to express this flexibility in the returned environment).
>>> >
>>> > Note
>>> > that we already give users control over the "main"
>>> environment via the
>>> > PortablePipelineOptions, so this wouldn't be an entirely
>>> new concept.
>>> >
>>> >
>>> > Yes, the author of a pipeline/transform chooses the
>>> environment in which
>>> > those transforms execute.
>>> >
>>> > The contrary position is that the Expansion Service
>>> should have full
>>> > control over which environment is chosen. Going back to
>>> the discussion
>>> > about artifact staging [2], this could enable to perform more
>>> > optimizations, such as merging environments or detecting
>>> conflicts.
>>> > However, this only works if this information has been
>>> provided upfront
>>> > to the Expansion Service. It wouldn't be impossible to
>>> provide these
>>> > hints alongside with the environment like suggested in
>>> the previous
>>> > paragraph.
>>> >
>>> > Any opinions? Should we allow users to optionally specify an
>>> > environment
>>> > for external transforms?
>>> >
>>> > Thanks,
>>> > Max
>>> >
>>> > [1] https://github.com/apache/beam/pull/8639
>>> > [2]
>>> >
>>> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>>> >
>>>
Re: Environments for External Transforms
Posted by Robert Bradshaw <ro...@google.com>.
On Mon, May 27, 2019 at 12:38 PM Maximilian Michels <mx...@apache.org> wrote:
>
> > Which environment would be used to perform the expansion? I think this is an interesting option, as long as it does not introduce a hard dependency on docker.
>
> The same environment that the to-be-expanded transform requires during
> runtime.
>
> > Dataflow has been doing something similar in this route where it is trying to get rid of the driver program running on the users machine. If you can get the expansion service to launch and run an environment to perform the expansion, you could also get it to create and submit a job as well returning data around the running job.
>
> Portability already runs without a driver on the user machine, apart
> from expansion and staging. For anything runtime-related the job server
> kicks in. It's worth to think about delegating expansion and staging to
> the Job server.
I think it makes a lot of sense for job servers to also act as
expansion services, but one can't of course defer expansion until job
submission.
> On 24.05.19 23:48, Lukasz Cwik wrote:
> > Dataflow has been doing something similar in this route where it is
> > trying to get rid of the driver program running on the users machine. If
> > you can get the expansion service to launch and run an environment to
> > perform the expansion, you could also get it to create and submit a job
> > as well returning data around the running job.
> >
> > On Thu, May 23, 2019 at 7:47 AM Thomas Weise <thw@apache.org
> > <ma...@apache.org>> wrote:
> >
> >
> >
> > On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > > Writing a new transform involves updating the expansion
> > service to include their new transform.
> >
> > Would it be conceivable that the expansion is performed via the
> > environment? That would solve the problem of updating the expansion
> > service, although it adds additional complexity for bringing up the
> > environment.
> >
> >
> > Which environment would be used to perform the expansion? I think
> > this is an interesting option, as long as it does not introduce a
> > hard dependency on docker.
> >
> > On 23.05.19 11:31, Robert Bradshaw wrote:
> > > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels
> > <mxm@apache.org <ma...@apache.org>
> > > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> > >
> > > Hi,
> > >
> > > Robert and me were discussing on the subject of
> > user-specified
> > > environments for external transforms [1]. We couldn't
> > decide whether
> > > users should have direct control over the environment
> > when they use an
> > > external transform in their pipeline.
> > >
> > > In my mind, it is quite natural that the Expansion
> > Service is a
> > > long-running service that gets started with a list of
> > available
> > > environments.
> > >
> > >
> > > +1.
> > >
> > > IMHO, the expansion service should be expected to provide valid
> > > environments for the transforms it vendors. Removing this
> > expectation
> > > seems wrong. Making it cheap to specify non-default
> > dependencies without
> > > building (publishing, etc.) a docker image is probably key to
> > making
> > > this work well (and also allowing more powerful environment
> > introspection).
> > >
> > > Such a list can be outdated and users may write transforms
> > > for a new environment they want to use in their pipeline.
> > >
> > >
> > > This is the part that I'm having trouble following. Writing a
> > new
> > > transform involves updating the expansion service to include
> > their new
> > > transform. The author of a transform (in other words, the one
> > who
> > > defines its expansion and implementation) is in the position
> > to name its
> > > dependencies, etc. and the user of the transform (the one
> > invoking it)
> > > is not in a generally good position to know what environments
> > would be
> > > valid.
> > >
> > > The easiest
> > > way would be to allow to pass the environment with the
> > transform.
> > >
> > >
> > > What this allows is using existing transforms in new
> > environments. There
> > > are possibly some usecases for this, e.g. expansion of a
> > given transform
> > > may be compatible with ether version X or version Y of a
> > library, left
> > > up to the discretion of the caller, but I think that this is
> > really just
> > > a deficiency in our environment specifications (e.g. it one
> > should be
> > > able to express this flexibility in the returned environment).
> > >
> > > Note
> > > that we already give users control over the "main"
> > environment via the
> > > PortablePipelineOptions, so this wouldn't be an entirely
> > new concept.
> > >
> > >
> > > Yes, the author of a pipeline/transform chooses the
> > environment in which
> > > those transforms execute.
> > >
> > > The contrary position is that the Expansion Service
> > should have full
> > > control over which environment is chosen. Going back to
> > the discussion
> > > about artifact staging [2], this could enable to perform more
> > > optimizations, such as merging environments or detecting
> > conflicts.
> > > However, this only works if this information has been
> > provided upfront
> > > to the Expansion Service. It wouldn't be impossible to
> > provide these
> > > hints alongside with the environment like suggested in
> > the previous
> > > paragraph.
> > >
> > > Any opinions? Should we allow users to optionally specify an
> > > environment
> > > for external transforms?
> > >
> > > Thanks,
> > > Max
> > >
> > > [1] https://github.com/apache/beam/pull/8639
> > > [2]
> > >
> > https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> > >
> >
Re: Environments for External Transforms
Posted by Maximilian Michels <mx...@apache.org>.
> Which environment would be used to perform the expansion? I think this is an interesting option, as long as it does not introduce a hard dependency on docker.
The same environment that the to-be-expanded transform requires during
runtime.
> Dataflow has been doing something similar in this route where it is trying to get rid of the driver program running on the users machine. If you can get the expansion service to launch and run an environment to perform the expansion, you could also get it to create and submit a job as well returning data around the running job.
Portability already runs without a driver on the user machine, apart
from expansion and staging. For anything runtime-related the job server
kicks in. It's worth to think about delegating expansion and staging to
the Job server.
On 24.05.19 23:48, Lukasz Cwik wrote:
> Dataflow has been doing something similar in this route where it is
> trying to get rid of the driver program running on the users machine. If
> you can get the expansion service to launch and run an environment to
> perform the expansion, you could also get it to create and submit a job
> as well returning data around the running job.
>
> On Thu, May 23, 2019 at 7:47 AM Thomas Weise <thw@apache.org
> <ma...@apache.org>> wrote:
>
>
>
> On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> > Writing a new transform involves updating the expansion
> service to include their new transform.
>
> Would it be conceivable that the expansion is performed via the
> environment? That would solve the problem of updating the expansion
> service, although it adds additional complexity for bringing up the
> environment.
>
>
> Which environment would be used to perform the expansion? I think
> this is an interesting option, as long as it does not introduce a
> hard dependency on docker.
>
> On 23.05.19 11:31, Robert Bradshaw wrote:
> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels
> <mxm@apache.org <ma...@apache.org>
> > <mailto:mxm@apache.org <ma...@apache.org>>> wrote:
> >
> > Hi,
> >
> > Robert and me were discussing on the subject of
> user-specified
> > environments for external transforms [1]. We couldn't
> decide whether
> > users should have direct control over the environment
> when they use an
> > external transform in their pipeline.
> >
> > In my mind, it is quite natural that the Expansion
> Service is a
> > long-running service that gets started with a list of
> available
> > environments.
> >
> >
> > +1.
> >
> > IMHO, the expansion service should be expected to provide valid
> > environments for the transforms it vendors. Removing this
> expectation
> > seems wrong. Making it cheap to specify non-default
> dependencies without
> > building (publishing, etc.) a docker image is probably key to
> making
> > this work well (and also allowing more powerful environment
> introspection).
> >
> > Such a list can be outdated and users may write transforms
> > for a new environment they want to use in their pipeline.
> >
> >
> > This is the part that I'm having trouble following. Writing a
> new
> > transform involves updating the expansion service to include
> their new
> > transform. The author of a transform (in other words, the one
> who
> > defines its expansion and implementation) is in the position
> to name its
> > dependencies, etc. and the user of the transform (the one
> invoking it)
> > is not in a generally good position to know what environments
> would be
> > valid.
> >
> > The easiest
> > way would be to allow to pass the environment with the
> transform.
> >
> >
> > What this allows is using existing transforms in new
> environments. There
> > are possibly some usecases for this, e.g. expansion of a
> given transform
> > may be compatible with ether version X or version Y of a
> library, left
> > up to the discretion of the caller, but I think that this is
> really just
> > a deficiency in our environment specifications (e.g. it one
> should be
> > able to express this flexibility in the returned environment).
> >
> > Note
> > that we already give users control over the "main"
> environment via the
> > PortablePipelineOptions, so this wouldn't be an entirely
> new concept.
> >
> >
> > Yes, the author of a pipeline/transform chooses the
> environment in which
> > those transforms execute.
> >
> > The contrary position is that the Expansion Service
> should have full
> > control over which environment is chosen. Going back to
> the discussion
> > about artifact staging [2], this could enable to perform more
> > optimizations, such as merging environments or detecting
> conflicts.
> > However, this only works if this information has been
> provided upfront
> > to the Expansion Service. It wouldn't be impossible to
> provide these
> > hints alongside with the environment like suggested in
> the previous
> > paragraph.
> >
> > Any opinions? Should we allow users to optionally specify an
> > environment
> > for external transforms?
> >
> > Thanks,
> > Max
> >
> > [1] https://github.com/apache/beam/pull/8639
> > [2]
> >
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> >
>
Re: Environments for External Transforms
Posted by Lukasz Cwik <lc...@google.com>.
Dataflow has been doing something similar in this route where it is trying
to get rid of the driver program running on the users machine. If you can
get the expansion service to launch and run an environment to perform the
expansion, you could also get it to create and submit a job as well
returning data around the running job.
On Thu, May 23, 2019 at 7:47 AM Thomas Weise <th...@apache.org> wrote:
>
>
> On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> > Writing a new transform involves updating the expansion service to
>> include their new transform.
>>
>> Would it be conceivable that the expansion is performed via the
>> environment? That would solve the problem of updating the expansion
>> service, although it adds additional complexity for bringing up the
>> environment.
>>
>>
> Which environment would be used to perform the expansion? I think this is
> an interesting option, as long as it does not introduce a hard dependency
> on docker.
>
>
>> On 23.05.19 11:31, Robert Bradshaw wrote:
>> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels <mxm@apache.org
>> > <ma...@apache.org>> wrote:
>> >
>> > Hi,
>> >
>> > Robert and me were discussing on the subject of user-specified
>> > environments for external transforms [1]. We couldn't decide whether
>> > users should have direct control over the environment when they use
>> an
>> > external transform in their pipeline.
>> >
>> > In my mind, it is quite natural that the Expansion Service is a
>> > long-running service that gets started with a list of available
>> > environments.
>> >
>> >
>> > +1.
>> >
>> > IMHO, the expansion service should be expected to provide valid
>> > environments for the transforms it vendors. Removing this expectation
>> > seems wrong. Making it cheap to specify non-default dependencies
>> without
>> > building (publishing, etc.) a docker image is probably key to making
>> > this work well (and also allowing more powerful environment
>> introspection).
>> >
>> > Such a list can be outdated and users may write transforms
>> > for a new environment they want to use in their pipeline.
>> >
>> >
>> > This is the part that I'm having trouble following. Writing a new
>> > transform involves updating the expansion service to include their new
>> > transform. The author of a transform (in other words, the one who
>> > defines its expansion and implementation) is in the position to name
>> its
>> > dependencies, etc. and the user of the transform (the one invoking it)
>> > is not in a generally good position to know what environments would be
>> > valid.
>> >
>> > The easiest
>> > way would be to allow to pass the environment with the transform.
>> >
>> >
>> > What this allows is using existing transforms in new environments.
>> There
>> > are possibly some usecases for this, e.g. expansion of a given
>> transform
>> > may be compatible with ether version X or version Y of a library, left
>> > up to the discretion of the caller, but I think that this is really
>> just
>> > a deficiency in our environment specifications (e.g. it one should be
>> > able to express this flexibility in the returned environment).
>> >
>> > Note
>> > that we already give users control over the "main" environment via
>> the
>> > PortablePipelineOptions, so this wouldn't be an entirely new
>> concept.
>> >
>> >
>> > Yes, the author of a pipeline/transform chooses the environment in
>> which
>> > those transforms execute.
>> >
>> > The contrary position is that the Expansion Service should have full
>> > control over which environment is chosen. Going back to the
>> discussion
>> > about artifact staging [2], this could enable to perform more
>> > optimizations, such as merging environments or detecting conflicts.
>> > However, this only works if this information has been provided
>> upfront
>> > to the Expansion Service. It wouldn't be impossible to provide these
>> > hints alongside with the environment like suggested in the previous
>> > paragraph.
>> >
>> > Any opinions? Should we allow users to optionally specify an
>> > environment
>> > for external transforms?
>> >
>> > Thanks,
>> > Max
>> >
>> > [1] https://github.com/apache/beam/pull/8639
>> > [2]
>> >
>> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>> >
>>
>
Re: Environments for External Transforms
Posted by Thomas Weise <th...@apache.org>.
On Thu, May 23, 2019 at 3:46 AM Maximilian Michels <mx...@apache.org> wrote:
> > Writing a new transform involves updating the expansion service to
> include their new transform.
>
> Would it be conceivable that the expansion is performed via the
> environment? That would solve the problem of updating the expansion
> service, although it adds additional complexity for bringing up the
> environment.
>
>
Which environment would be used to perform the expansion? I think this is
an interesting option, as long as it does not introduce a hard dependency
on docker.
> On 23.05.19 11:31, Robert Bradshaw wrote:
> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi,
> >
> > Robert and me were discussing on the subject of user-specified
> > environments for external transforms [1]. We couldn't decide whether
> > users should have direct control over the environment when they use
> an
> > external transform in their pipeline.
> >
> > In my mind, it is quite natural that the Expansion Service is a
> > long-running service that gets started with a list of available
> > environments.
> >
> >
> > +1.
> >
> > IMHO, the expansion service should be expected to provide valid
> > environments for the transforms it vendors. Removing this expectation
> > seems wrong. Making it cheap to specify non-default dependencies without
> > building (publishing, etc.) a docker image is probably key to making
> > this work well (and also allowing more powerful environment
> introspection).
> >
> > Such a list can be outdated and users may write transforms
> > for a new environment they want to use in their pipeline.
> >
> >
> > This is the part that I'm having trouble following. Writing a new
> > transform involves updating the expansion service to include their new
> > transform. The author of a transform (in other words, the one who
> > defines its expansion and implementation) is in the position to name its
> > dependencies, etc. and the user of the transform (the one invoking it)
> > is not in a generally good position to know what environments would be
> > valid.
> >
> > The easiest
> > way would be to allow to pass the environment with the transform.
> >
> >
> > What this allows is using existing transforms in new environments. There
> > are possibly some usecases for this, e.g. expansion of a given transform
> > may be compatible with ether version X or version Y of a library, left
> > up to the discretion of the caller, but I think that this is really just
> > a deficiency in our environment specifications (e.g. it one should be
> > able to express this flexibility in the returned environment).
> >
> > Note
> > that we already give users control over the "main" environment via
> the
> > PortablePipelineOptions, so this wouldn't be an entirely new concept.
> >
> >
> > Yes, the author of a pipeline/transform chooses the environment in which
> > those transforms execute.
> >
> > The contrary position is that the Expansion Service should have full
> > control over which environment is chosen. Going back to the
> discussion
> > about artifact staging [2], this could enable to perform more
> > optimizations, such as merging environments or detecting conflicts.
> > However, this only works if this information has been provided
> upfront
> > to the Expansion Service. It wouldn't be impossible to provide these
> > hints alongside with the environment like suggested in the previous
> > paragraph.
> >
> > Any opinions? Should we allow users to optionally specify an
> > environment
> > for external transforms?
> >
> > Thanks,
> > Max
> >
> > [1] https://github.com/apache/beam/pull/8639
> > [2]
> >
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> >
>
Re: Environments for External Transforms
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, May 23, 2019 at 12:46 PM Maximilian Michels <mx...@apache.org> wrote:
> > Writing a new transform involves updating the expansion service to
> include their new transform.
>
> Would it be conceivable that the expansion is performed via the
> environment? That would solve the problem of updating the expansion
> service, although it adds additional complexity for bringing up the
> environment.
>
I think it's makes sense to provide docker images that can be run as both
expansion services and environments. I could also imagine a service that
takes an environment in which to perform the expansion (though this would
probably not be offered by most, let alone all, expansion services).
> On 23.05.19 11:31, Robert Bradshaw wrote:
> > On Wed, May 22, 2019 at 6:17 PM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi,
> >
> > Robert and me were discussing on the subject of user-specified
> > environments for external transforms [1]. We couldn't decide whether
> > users should have direct control over the environment when they use
> an
> > external transform in their pipeline.
> >
> > In my mind, it is quite natural that the Expansion Service is a
> > long-running service that gets started with a list of available
> > environments.
> >
> >
> > +1.
> >
> > IMHO, the expansion service should be expected to provide valid
> > environments for the transforms it vendors. Removing this expectation
> > seems wrong. Making it cheap to specify non-default dependencies without
> > building (publishing, etc.) a docker image is probably key to making
> > this work well (and also allowing more powerful environment
> introspection).
> >
> > Such a list can be outdated and users may write transforms
> > for a new environment they want to use in their pipeline.
> >
> >
> > This is the part that I'm having trouble following. Writing a new
> > transform involves updating the expansion service to include their new
> > transform. The author of a transform (in other words, the one who
> > defines its expansion and implementation) is in the position to name its
> > dependencies, etc. and the user of the transform (the one invoking it)
> > is not in a generally good position to know what environments would be
> > valid.
> >
> > The easiest
> > way would be to allow to pass the environment with the transform.
> >
> >
> > What this allows is using existing transforms in new environments. There
> > are possibly some usecases for this, e.g. expansion of a given transform
> > may be compatible with ether version X or version Y of a library, left
> > up to the discretion of the caller, but I think that this is really just
> > a deficiency in our environment specifications (e.g. it one should be
> > able to express this flexibility in the returned environment).
> >
> > Note
> > that we already give users control over the "main" environment via
> the
> > PortablePipelineOptions, so this wouldn't be an entirely new concept.
> >
> >
> > Yes, the author of a pipeline/transform chooses the environment in which
> > those transforms execute.
> >
> > The contrary position is that the Expansion Service should have full
> > control over which environment is chosen. Going back to the
> discussion
> > about artifact staging [2], this could enable to perform more
> > optimizations, such as merging environments or detecting conflicts.
> > However, this only works if this information has been provided
> upfront
> > to the Expansion Service. It wouldn't be impossible to provide these
> > hints alongside with the environment like suggested in the previous
> > paragraph.
> >
> > Any opinions? Should we allow users to optionally specify an
> > environment
> > for external transforms?
> >
> > Thanks,
> > Max
> >
> > [1] https://github.com/apache/beam/pull/8639
> > [2]
> >
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> >
>
Re: Environments for External Transforms
Posted by Maximilian Michels <mx...@apache.org>.
> Writing a new transform involves updating the expansion service to include their new transform.
Would it be conceivable that the expansion is performed via the
environment? That would solve the problem of updating the expansion
service, although it adds additional complexity for bringing up the
environment.
On 23.05.19 11:31, Robert Bradshaw wrote:
> On Wed, May 22, 2019 at 6:17 PM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi,
>
> Robert and me were discussing on the subject of user-specified
> environments for external transforms [1]. We couldn't decide whether
> users should have direct control over the environment when they use an
> external transform in their pipeline.
>
> In my mind, it is quite natural that the Expansion Service is a
> long-running service that gets started with a list of available
> environments.
>
>
> +1.
>
> IMHO, the expansion service should be expected to provide valid
> environments for the transforms it vendors. Removing this expectation
> seems wrong. Making it cheap to specify non-default dependencies without
> building (publishing, etc.) a docker image is probably key to making
> this work well (and also allowing more powerful environment introspection).
>
> Such a list can be outdated and users may write transforms
> for a new environment they want to use in their pipeline.
>
>
> This is the part that I'm having trouble following. Writing a new
> transform involves updating the expansion service to include their new
> transform. The author of a transform (in other words, the one who
> defines its expansion and implementation) is in the position to name its
> dependencies, etc. and the user of the transform (the one invoking it)
> is not in a generally good position to know what environments would be
> valid.
>
> The easiest
> way would be to allow to pass the environment with the transform.
>
>
> What this allows is using existing transforms in new environments. There
> are possibly some usecases for this, e.g. expansion of a given transform
> may be compatible with ether version X or version Y of a library, left
> up to the discretion of the caller, but I think that this is really just
> a deficiency in our environment specifications (e.g. it one should be
> able to express this flexibility in the returned environment).
>
> Note
> that we already give users control over the "main" environment via the
> PortablePipelineOptions, so this wouldn't be an entirely new concept.
>
>
> Yes, the author of a pipeline/transform chooses the environment in which
> those transforms execute.
>
> The contrary position is that the Expansion Service should have full
> control over which environment is chosen. Going back to the discussion
> about artifact staging [2], this could enable to perform more
> optimizations, such as merging environments or detecting conflicts.
> However, this only works if this information has been provided upfront
> to the Expansion Service. It wouldn't be impossible to provide these
> hints alongside with the environment like suggested in the previous
> paragraph.
>
> Any opinions? Should we allow users to optionally specify an
> environment
> for external transforms?
>
> Thanks,
> Max
>
> [1] https://github.com/apache/beam/pull/8639
> [2]
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>
Re: Environments for External Transforms
Posted by Robert Bradshaw <ro...@google.com>.
On Wed, May 22, 2019 at 6:17 PM Maximilian Michels <mx...@apache.org> wrote:
> Hi,
>
> Robert and me were discussing on the subject of user-specified
> environments for external transforms [1]. We couldn't decide whether
> users should have direct control over the environment when they use an
> external transform in their pipeline.
>
> In my mind, it is quite natural that the Expansion Service is a
> long-running service that gets started with a list of available
> environments.
+1.
IMHO, the expansion service should be expected to provide valid
environments for the transforms it vendors. Removing this expectation seems
wrong. Making it cheap to specify non-default dependencies without building
(publishing, etc.) a docker image is probably key to making this work well
(and also allowing more powerful environment introspection).
> Such a list can be outdated and users may write transforms
> for a new environment they want to use in their pipeline.
This is the part that I'm having trouble following. Writing a new transform
involves updating the expansion service to include their new transform. The
author of a transform (in other words, the one who defines its expansion
and implementation) is in the position to name its dependencies, etc. and
the user of the transform (the one invoking it) is not in a generally good
position to know what environments would be valid.
> The easiest
> way would be to allow to pass the environment with the transform.
What this allows is using existing transforms in new environments. There
are possibly some usecases for this, e.g. expansion of a given transform
may be compatible with ether version X or version Y of a library, left up
to the discretion of the caller, but I think that this is really just a
deficiency in our environment specifications (e.g. it one should be able to
express this flexibility in the returned environment).
> Note
> that we already give users control over the "main" environment via the
> PortablePipelineOptions, so this wouldn't be an entirely new concept.
>
Yes, the author of a pipeline/transform chooses the environment in which
those transforms execute.
> The contrary position is that the Expansion Service should have full
> control over which environment is chosen. Going back to the discussion
> about artifact staging [2], this could enable to perform more
> optimizations, such as merging environments or detecting conflicts.
> However, this only works if this information has been provided upfront
> to the Expansion Service. It wouldn't be impossible to provide these
> hints alongside with the environment like suggested in the previous
> paragraph.
>
> Any opinions? Should we allow users to optionally specify an environment
> for external transforms?
>
> Thanks,
> Max
>
> [1] https://github.com/apache/beam/pull/8639
> [2]
>
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>
Re: Environments for External Transforms
Posted by Robert Bradshaw <ro...@google.com>.
On Thu, May 23, 2019 at 11:07 AM Maximilian Michels <mx...@apache.org> wrote:
> My motivation was to get rid of the Docker dependency for the Python VR
> tests. Similarly to how we use Python's LOOPBACK environment for
> executing all non-cross-language tests, I wanted to use Java's EMBEDDED
> environment to run the cross-language transforms.
>
> I suppose we could also go with an override of the default environment
> during startup of the Expansion Service. This would be less intrusive
> and still allow us to use the embedded environment in some of the tests.
>
Yes, I think for the purpose of cheap testing customizing the expansion
service to provide embedded environments is a good solution.
Long term, I think the answer is the ability for a runner to
intelligently swap out (merge, reconcile) environments in a compatible way,
e.g. a runner could be instructed to use the embedded environment instead
as a cheap replacement whenever the standard java environment is
encountered. This would allow one to write pipelines that (for example)
embed the java portions when running on a java runner, the python portions
when running on a python runner, and neither when running on a C++ runner
without hard-coding expectations at the expansion call sites. I would
imagine runners could even be conservative about which environments they
choose to run in embedded modes, depending on whether the specified
dependencies are compatible with their own.
> > 2(c) can also be "hacked" inside an SDK as an explicit environment
> override by the "user" where the expansion service isn't involved and the
> user/SDK manipulates the expansion service response. As Chamikara pointed
> out, I believe the response from the expansion service should be "safe"
> instead of allowing it to return broken combinations.
>
> The responses are not safe today because we do not have a check whether
> an expanded transform is compatible with the default environment. I
> agree though that it is better to develop these checks first before
> allowing overrides by the user.
>
> I'll need to follow-up with the artifact staging doc, then we can agree
> on the first steps for implementation.
>
> Thanks,
> Max
>
> On 22.05.19 20:13, Lukasz Cwik wrote:
> > 2(c) can also be "hacked" inside an SDK as an explicit environment
> > override by the "user" where the expansion service isn't involved and
> > the user/SDK manipulates the expansion service response. As Chamikara
> > pointed out, I believe the response from the expansion service should be
> > "safe" instead of allowing it to return broken combinations.
> >
> > On Wed, May 22, 2019 at 11:08 AM Chamikara Jayalath
> > <chamikara@google.com <ma...@google.com>> wrote:
> >
> >
> >
> > On Wed, May 22, 2019 at 9:17 AM Maximilian Michels <mxm@apache.org
> > <ma...@apache.org>> wrote:
> >
> > Hi,
> >
> > Robert and me were discussing on the subject of user-specified
> > environments for external transforms [1]. We couldn't decide
> > whether
> > users should have direct control over the environment when they
> > use an
> > external transform in their pipeline.
> >
> > In my mind, it is quite natural that the Expansion Service is a
> > long-running service that gets started with a list of available
> > environments. Such a list can be outdated and users may write
> > transforms
> > for a new environment they want to use in their pipeline. The
> > easiest
> > way would be to allow to pass the environment with the
> > transform. Note
> > that we already give users control over the "main" environment
> > via the
> > PortablePipelineOptions, so this wouldn't be an entirely new
> > concept.
> >
> >
> >
> > I think we are trying to generalize the expansion service along
> > multiple axes.
> > (1) dependencies
> > (a) dependencies embedded in an environment (b) dependencies
> > specific to an transform (c) dependencies specified by the user
> > expanding the transform
> >
> > (2) environments
> > (a)default environment (b) environments specified a startup of the
> > expansion service (c) environments specified by the user expanding
> > the transform (this proposal)
> >
> > It's great if we can implement the most generic solution along all
> > these exes but I think we run into risk of resulting in broken
> > combinations by trying to implement this before we have other
> > necessary pieces to support a long running expansion service. For
> > example, support for dynamically registering transforms and support
> > for discovering transforms.
> >
> > What is the need for implementing 2 (c) now ? If there's no real
> > need now I suggest we settle with 2(a) or 2(b) for now till we can
> > truly support a long running expansion service. Also we'll have a
> > better idea of how this kind if features should evolve when we have
> > at least two runners supporting cross-language transforms (we are in
> > the process of updating Dataflow to support this). Just my 2 cents
> > though :)
> >
> >
> > The contrary position is that the Expansion Service should have
> > full
> > control over which environment is chosen. Going back to the
> > discussion
> > about artifact staging [2], this could enable to perform more
> > optimizations, such as merging environments or detecting
> conflicts.
> > However, this only works if this information has been provided
> > upfront
> > to the Expansion Service. It wouldn't be impossible to provide
> > these
> > hints alongside with the environment like suggested in the
> previous
> > paragraph.
> >
> > Any opinions? Should we allow users to optionally specify an
> > environment
> > for external transforms?
> >
> > Thanks,
> > Max
> >
> > [1] https://github.com/apache/beam/pull/8639
> > [2]
> >
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
> >
>
Re: Environments for External Transforms
Posted by Maximilian Michels <mx...@apache.org>.
My motivation was to get rid of the Docker dependency for the Python VR
tests. Similarly to how we use Python's LOOPBACK environment for
executing all non-cross-language tests, I wanted to use Java's EMBEDDED
environment to run the cross-language transforms.
I suppose we could also go with an override of the default environment
during startup of the Expansion Service. This would be less intrusive
and still allow us to use the embedded environment in some of the tests.
> 2(c) can also be "hacked" inside an SDK as an explicit environment override by the "user" where the expansion service isn't involved and the user/SDK manipulates the expansion service response. As Chamikara pointed out, I believe the response from the expansion service should be "safe" instead of allowing it to return broken combinations.
The responses are not safe today because we do not have a check whether
an expanded transform is compatible with the default environment. I
agree though that it is better to develop these checks first before
allowing overrides by the user.
I'll need to follow-up with the artifact staging doc, then we can agree
on the first steps for implementation.
Thanks,
Max
On 22.05.19 20:13, Lukasz Cwik wrote:
> 2(c) can also be "hacked" inside an SDK as an explicit environment
> override by the "user" where the expansion service isn't involved and
> the user/SDK manipulates the expansion service response. As Chamikara
> pointed out, I believe the response from the expansion service should be
> "safe" instead of allowing it to return broken combinations.
>
> On Wed, May 22, 2019 at 11:08 AM Chamikara Jayalath
> <chamikara@google.com <ma...@google.com>> wrote:
>
>
>
> On Wed, May 22, 2019 at 9:17 AM Maximilian Michels <mxm@apache.org
> <ma...@apache.org>> wrote:
>
> Hi,
>
> Robert and me were discussing on the subject of user-specified
> environments for external transforms [1]. We couldn't decide
> whether
> users should have direct control over the environment when they
> use an
> external transform in their pipeline.
>
> In my mind, it is quite natural that the Expansion Service is a
> long-running service that gets started with a list of available
> environments. Such a list can be outdated and users may write
> transforms
> for a new environment they want to use in their pipeline. The
> easiest
> way would be to allow to pass the environment with the
> transform. Note
> that we already give users control over the "main" environment
> via the
> PortablePipelineOptions, so this wouldn't be an entirely new
> concept.
>
>
>
> I think we are trying to generalize the expansion service along
> multiple axes.
> (1) dependencies
> (a) dependencies embedded in an environment (b) dependencies
> specific to an transform (c) dependencies specified by the user
> expanding the transform
>
> (2) environments
> (a)default environment (b) environments specified a startup of the
> expansion service (c) environments specified by the user expanding
> the transform (this proposal)
>
> It's great if we can implement the most generic solution along all
> these exes but I think we run into risk of resulting in broken
> combinations by trying to implement this before we have other
> necessary pieces to support a long running expansion service. For
> example, support for dynamically registering transforms and support
> for discovering transforms.
>
> What is the need for implementing 2 (c) now ? If there's no real
> need now I suggest we settle with 2(a) or 2(b) for now till we can
> truly support a long running expansion service. Also we'll have a
> better idea of how this kind if features should evolve when we have
> at least two runners supporting cross-language transforms (we are in
> the process of updating Dataflow to support this). Just my 2 cents
> though :)
>
>
> The contrary position is that the Expansion Service should have
> full
> control over which environment is chosen. Going back to the
> discussion
> about artifact staging [2], this could enable to perform more
> optimizations, such as merging environments or detecting conflicts.
> However, this only works if this information has been provided
> upfront
> to the Expansion Service. It wouldn't be impossible to provide
> these
> hints alongside with the environment like suggested in the previous
> paragraph.
>
> Any opinions? Should we allow users to optionally specify an
> environment
> for external transforms?
>
> Thanks,
> Max
>
> [1] https://github.com/apache/beam/pull/8639
> [2]
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>
Re: Environments for External Transforms
Posted by Lukasz Cwik <lc...@google.com>.
2(c) can also be "hacked" inside an SDK as an explicit environment override
by the "user" where the expansion service isn't involved and the user/SDK
manipulates the expansion service response. As Chamikara pointed out, I
believe the response from the expansion service should be "safe" instead of
allowing it to return broken combinations.
On Wed, May 22, 2019 at 11:08 AM Chamikara Jayalath <ch...@google.com>
wrote:
>
>
> On Wed, May 22, 2019 at 9:17 AM Maximilian Michels <mx...@apache.org> wrote:
>
>> Hi,
>>
>> Robert and me were discussing on the subject of user-specified
>> environments for external transforms [1]. We couldn't decide whether
>> users should have direct control over the environment when they use an
>> external transform in their pipeline.
>>
>> In my mind, it is quite natural that the Expansion Service is a
>> long-running service that gets started with a list of available
>> environments. Such a list can be outdated and users may write transforms
>> for a new environment they want to use in their pipeline. The easiest
>> way would be to allow to pass the environment with the transform. Note
>> that we already give users control over the "main" environment via the
>> PortablePipelineOptions, so this wouldn't be an entirely new concept.
>>
>
>
> I think we are trying to generalize the expansion service along multiple
> axes.
> (1) dependencies
> (a) dependencies embedded in an environment (b) dependencies specific to
> an transform (c) dependencies specified by the user expanding the transform
>
> (2) environments
> (a)default environment (b) environments specified a startup of the
> expansion service (c) environments specified by the user expanding the
> transform (this proposal)
>
> It's great if we can implement the most generic solution along all these
> exes but I think we run into risk of resulting in broken combinations by
> trying to implement this before we have other necessary pieces to support a
> long running expansion service. For example, support for dynamically
> registering transforms and support for discovering transforms.
>
> What is the need for implementing 2 (c) now ? If there's no real need now
> I suggest we settle with 2(a) or 2(b) for now till we can truly support a
> long running expansion service. Also we'll have a better idea of how this
> kind if features should evolve when we have at least two runners supporting
> cross-language transforms (we are in the process of updating Dataflow to
> support this). Just my 2 cents though :)
>
>
>>
>> The contrary position is that the Expansion Service should have full
>> control over which environment is chosen. Going back to the discussion
>> about artifact staging [2], this could enable to perform more
>> optimizations, such as merging environments or detecting conflicts.
>> However, this only works if this information has been provided upfront
>> to the Expansion Service. It wouldn't be impossible to provide these
>> hints alongside with the environment like suggested in the previous
>> paragraph.
>>
>> Any opinions? Should we allow users to optionally specify an environment
>> for external transforms?
>>
>> Thanks,
>> Max
>>
>> [1] https://github.com/apache/beam/pull/8639
>> [2]
>>
>> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>>
>
Re: Environments for External Transforms
Posted by Chamikara Jayalath <ch...@google.com>.
On Wed, May 22, 2019 at 9:17 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi,
>
> Robert and me were discussing on the subject of user-specified
> environments for external transforms [1]. We couldn't decide whether
> users should have direct control over the environment when they use an
> external transform in their pipeline.
>
> In my mind, it is quite natural that the Expansion Service is a
> long-running service that gets started with a list of available
> environments. Such a list can be outdated and users may write transforms
> for a new environment they want to use in their pipeline. The easiest
> way would be to allow to pass the environment with the transform. Note
> that we already give users control over the "main" environment via the
> PortablePipelineOptions, so this wouldn't be an entirely new concept.
>
I think we are trying to generalize the expansion service along multiple
axes.
(1) dependencies
(a) dependencies embedded in an environment (b) dependencies specific to an
transform (c) dependencies specified by the user expanding the transform
(2) environments
(a)default environment (b) environments specified a startup of the
expansion service (c) environments specified by the user expanding the
transform (this proposal)
It's great if we can implement the most generic solution along all these
exes but I think we run into risk of resulting in broken combinations by
trying to implement this before we have other necessary pieces to support a
long running expansion service. For example, support for dynamically
registering transforms and support for discovering transforms.
What is the need for implementing 2 (c) now ? If there's no real need now I
suggest we settle with 2(a) or 2(b) for now till we can truly support a
long running expansion service. Also we'll have a better idea of how this
kind if features should evolve when we have at least two runners supporting
cross-language transforms (we are in the process of updating Dataflow to
support this). Just my 2 cents though :)
>
> The contrary position is that the Expansion Service should have full
> control over which environment is chosen. Going back to the discussion
> about artifact staging [2], this could enable to perform more
> optimizations, such as merging environments or detecting conflicts.
> However, this only works if this information has been provided upfront
> to the Expansion Service. It wouldn't be impossible to provide these
> hints alongside with the environment like suggested in the previous
> paragraph.
>
> Any opinions? Should we allow users to optionally specify an environment
> for external transforms?
>
> Thanks,
> Max
>
> [1] https://github.com/apache/beam/pull/8639
> [2]
>
> https://lists.apache.org/thread.html/6fcee7047f53cf1c0636fb65367ef70842016d57effe2e5795c4137d@%3Cdev.beam.apache.org%3E
>