You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Steve Niemitz <sn...@apache.org> on 2021/10/06 13:35:44 UTC

ExpansionService...as a service?

Has anyone ever tried hosting a long-running expansion service as a real
"service", the intent being that users don't need to run it locally, and
can instead connect to the shared one when expanding pipelines?

Looking around the code I already see a few assumptions that it will only
live for a short while (memory leaks, etc), as well as some pretty large
security concerns (eg, the artifact retrieval service can be used to
download any file on the host) curious if anyone has put any work into this
already?

Re: ExpansionService...as a service?

Posted by Steve Niemitz <sn...@apache.org>.
> I was imagining a long running expansion service would just serve up a
fixed number of transforms, and changing this set would be done via a new
release of the service.

This is how I was thinking about it as well, I don't think we'd have any
requirement to dynamically add/remove transforms at runtime.

On Wed, Oct 6, 2021 at 1:27 PM Robert Bradshaw <ro...@google.com> wrote:

> On Wed, Oct 6, 2021 at 10:24 AM Chamikara Jayalath <ch...@google.com>
> wrote:
> >
> > Regarding the memory leak, I'm not sure if a function registration is
> needed here since it seems like that function is doing something pretty
> generic. +Heejong Lee should know more.
>
> +1
>
> > Regarding a long running expansion service, in addition to what others
> mentioned, I think we also need protocols for dynamically
> registering/unregistering transforms and a protocol for discovering
> transforms already registered in an expansion service.
>
> I was imagining a long running expansion service would just serve up a
> fixed number of transforms, and changing this set would be done via a
> new release of the service.
>
> Having a dynamic set of transforms served (e.g. expansion services as
> a service) is another level that could be interesting to explore
> (though more complicated, e.g. now you're hosting third party jars and
> running arbitrary code).
>
> > On Wed, Oct 6, 2021 at 10:02 AM Steve Niemitz <sn...@apache.org>
> wrote:
> >>
> >> I noticed that Environments.getDeferredArtifacts will add a function to
> DefaultArtifactResolver.INSTANCE which is never removed, so those will
> accumulate forever.  That's the only one I noticed so far from code review,
> but it's possible there's more.
> >>
> >> On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <ro...@google.com>
> wrote:
> >>>
> >>> IIRC, the folks at Lyft offered this as a service (internally, not
> >>> publicly). We've toyed with the idea of offering it more generally.
> >>> You're right that there would be security implications (though some
> >>> generic sandboxing could go a long way). What were the memory leaks
> >>> that you were noticing?
> >>>
> >>> On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <sn...@apache.org>
> wrote:
> >>> >
> >>> > cool, thanks for the info.  I might be the first to try then :)
> >>> >
> >>> > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:
> >>> >>
> >>> >> I believe that was one of the ideas behind an expansion service but
> to my knowledge I don't know of anyone who has a long running expansion
> service. I was thinking that the Apache Beam community at some point could
> host one so that users don't need to spin up Java/Python/Go expansion
> services to run a subset of XLang pipelines.
> >>> >>
> >>> >> The XLang effort has been focusing about getting more transforms
> working and solving ease of use issues for users so I don't believe what
> you describe has been done.
> >>> >>
> >>> >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com>
> wrote:
> >>> >>>
> >>> >>> /cc @Chamikara Jayalath @Robert Bradshaw
> >>> >>>
> >>> >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org>
> wrote:
> >>> >>>>
> >>> >>>> Has anyone ever tried hosting a long-running expansion service as
> a real "service", the intent being that users don't need to run it locally,
> and can instead connect to the shared one when expanding pipelines?
> >>> >>>>
> >>> >>>> Looking around the code I already see a few assumptions that it
> will only live for a short while (memory leaks, etc), as well as some
> pretty large security concerns (eg, the artifact retrieval service can be
> used to download any file on the host) curious if anyone has put any work
> into this already?
>

Re: ExpansionService...as a service?

Posted by Robert Bradshaw <ro...@google.com>.
On Wed, Oct 6, 2021 at 10:24 AM Chamikara Jayalath <ch...@google.com> wrote:
>
> Regarding the memory leak, I'm not sure if a function registration is needed here since it seems like that function is doing something pretty generic. +Heejong Lee should know more.

+1

> Regarding a long running expansion service, in addition to what others mentioned, I think we also need protocols for dynamically registering/unregistering transforms and a protocol for discovering transforms already registered in an expansion service.

I was imagining a long running expansion service would just serve up a
fixed number of transforms, and changing this set would be done via a
new release of the service.

Having a dynamic set of transforms served (e.g. expansion services as
a service) is another level that could be interesting to explore
(though more complicated, e.g. now you're hosting third party jars and
running arbitrary code).

> On Wed, Oct 6, 2021 at 10:02 AM Steve Niemitz <sn...@apache.org> wrote:
>>
>> I noticed that Environments.getDeferredArtifacts will add a function to DefaultArtifactResolver.INSTANCE which is never removed, so those will accumulate forever.  That's the only one I noticed so far from code review, but it's possible there's more.
>>
>> On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <ro...@google.com> wrote:
>>>
>>> IIRC, the folks at Lyft offered this as a service (internally, not
>>> publicly). We've toyed with the idea of offering it more generally.
>>> You're right that there would be security implications (though some
>>> generic sandboxing could go a long way). What were the memory leaks
>>> that you were noticing?
>>>
>>> On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <sn...@apache.org> wrote:
>>> >
>>> > cool, thanks for the info.  I might be the first to try then :)
>>> >
>>> > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:
>>> >>
>>> >> I believe that was one of the ideas behind an expansion service but to my knowledge I don't know of anyone who has a long running expansion service. I was thinking that the Apache Beam community at some point could host one so that users don't need to spin up Java/Python/Go expansion services to run a subset of XLang pipelines.
>>> >>
>>> >> The XLang effort has been focusing about getting more transforms working and solving ease of use issues for users so I don't believe what you describe has been done.
>>> >>
>>> >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:
>>> >>>
>>> >>> /cc @Chamikara Jayalath @Robert Bradshaw
>>> >>>
>>> >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org> wrote:
>>> >>>>
>>> >>>> Has anyone ever tried hosting a long-running expansion service as a real "service", the intent being that users don't need to run it locally, and can instead connect to the shared one when expanding pipelines?
>>> >>>>
>>> >>>> Looking around the code I already see a few assumptions that it will only live for a short while (memory leaks, etc), as well as some pretty large security concerns (eg, the artifact retrieval service can be used to download any file on the host) curious if anyone has put any work into this already?

Re: ExpansionService...as a service?

Posted by Chamikara Jayalath <ch...@google.com>.
Regarding the memory leak, I'm not sure if a function registration is
needed here since it seems like that function is doing something pretty
generic. +Heejong Lee <he...@google.com> should know more.

Regarding a long running expansion service, in addition to what others
mentioned, I think we also need protocols for dynamically
registering/unregistering transforms and a protocol for discovering
transforms already registered in an expansion service.

Thanks,
Cham


On Wed, Oct 6, 2021 at 10:02 AM Steve Niemitz <sn...@apache.org> wrote:

> I noticed that Environments.getDeferredArtifacts will add a function
> to DefaultArtifactResolver.INSTANCE which is never removed, so those will
> accumulate forever.  That's the only one I noticed so far from code review,
> but it's possible there's more.
>
> On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> IIRC, the folks at Lyft offered this as a service (internally, not
>> publicly). We've toyed with the idea of offering it more generally.
>> You're right that there would be security implications (though some
>> generic sandboxing could go a long way). What were the memory leaks
>> that you were noticing?
>>
>> On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <sn...@apache.org> wrote:
>> >
>> > cool, thanks for the info.  I might be the first to try then :)
>> >
>> > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:
>> >>
>> >> I believe that was one of the ideas behind an expansion service but to
>> my knowledge I don't know of anyone who has a long running expansion
>> service. I was thinking that the Apache Beam community at some point could
>> host one so that users don't need to spin up Java/Python/Go expansion
>> services to run a subset of XLang pipelines.
>> >>
>> >> The XLang effort has been focusing about getting more transforms
>> working and solving ease of use issues for users so I don't believe what
>> you describe has been done.
>> >>
>> >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:
>> >>>
>> >>> /cc @Chamikara Jayalath @Robert Bradshaw
>> >>>
>> >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org>
>> wrote:
>> >>>>
>> >>>> Has anyone ever tried hosting a long-running expansion service as a
>> real "service", the intent being that users don't need to run it locally,
>> and can instead connect to the shared one when expanding pipelines?
>> >>>>
>> >>>> Looking around the code I already see a few assumptions that it will
>> only live for a short while (memory leaks, etc), as well as some pretty
>> large security concerns (eg, the artifact retrieval service can be used to
>> download any file on the host) curious if anyone has put any work into this
>> already?
>>
>

Re: ExpansionService...as a service?

Posted by Steve Niemitz <sn...@apache.org>.
I noticed that Environments.getDeferredArtifacts will add a function
to DefaultArtifactResolver.INSTANCE which is never removed, so those will
accumulate forever.  That's the only one I noticed so far from code review,
but it's possible there's more.

On Wed, Oct 6, 2021 at 12:38 PM Robert Bradshaw <ro...@google.com> wrote:

> IIRC, the folks at Lyft offered this as a service (internally, not
> publicly). We've toyed with the idea of offering it more generally.
> You're right that there would be security implications (though some
> generic sandboxing could go a long way). What were the memory leaks
> that you were noticing?
>
> On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <sn...@apache.org> wrote:
> >
> > cool, thanks for the info.  I might be the first to try then :)
> >
> > On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:
> >>
> >> I believe that was one of the ideas behind an expansion service but to
> my knowledge I don't know of anyone who has a long running expansion
> service. I was thinking that the Apache Beam community at some point could
> host one so that users don't need to spin up Java/Python/Go expansion
> services to run a subset of XLang pipelines.
> >>
> >> The XLang effort has been focusing about getting more transforms
> working and solving ease of use issues for users so I don't believe what
> you describe has been done.
> >>
> >> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:
> >>>
> >>> /cc @Chamikara Jayalath @Robert Bradshaw
> >>>
> >>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org>
> wrote:
> >>>>
> >>>> Has anyone ever tried hosting a long-running expansion service as a
> real "service", the intent being that users don't need to run it locally,
> and can instead connect to the shared one when expanding pipelines?
> >>>>
> >>>> Looking around the code I already see a few assumptions that it will
> only live for a short while (memory leaks, etc), as well as some pretty
> large security concerns (eg, the artifact retrieval service can be used to
> download any file on the host) curious if anyone has put any work into this
> already?
>

Re: ExpansionService...as a service?

Posted by Robert Bradshaw <ro...@google.com>.
IIRC, the folks at Lyft offered this as a service (internally, not
publicly). We've toyed with the idea of offering it more generally.
You're right that there would be security implications (though some
generic sandboxing could go a long way). What were the memory leaks
that you were noticing?

On Wed, Oct 6, 2021 at 9:13 AM Steve Niemitz <sn...@apache.org> wrote:
>
> cool, thanks for the info.  I might be the first to try then :)
>
> On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:
>>
>> I believe that was one of the ideas behind an expansion service but to my knowledge I don't know of anyone who has a long running expansion service. I was thinking that the Apache Beam community at some point could host one so that users don't need to spin up Java/Python/Go expansion services to run a subset of XLang pipelines.
>>
>> The XLang effort has been focusing about getting more transforms working and solving ease of use issues for users so I don't believe what you describe has been done.
>>
>> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:
>>>
>>> /cc @Chamikara Jayalath @Robert Bradshaw
>>>
>>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org> wrote:
>>>>
>>>> Has anyone ever tried hosting a long-running expansion service as a real "service", the intent being that users don't need to run it locally, and can instead connect to the shared one when expanding pipelines?
>>>>
>>>> Looking around the code I already see a few assumptions that it will only live for a short while (memory leaks, etc), as well as some pretty large security concerns (eg, the artifact retrieval service can be used to download any file on the host) curious if anyone has put any work into this already?

Re: ExpansionService...as a service?

Posted by Steve Niemitz <sn...@apache.org>.
cool, thanks for the info.  I might be the first to try then :)

On Wed, Oct 6, 2021 at 12:00 PM Luke Cwik <lc...@google.com> wrote:

> I believe that was one of the ideas behind an expansion service but to my
> knowledge I don't know of anyone who has a long running expansion service.
> I was thinking that the Apache Beam community at some point could host one
> so that users don't need to spin up Java/Python/Go expansion services to
> run a subset of XLang pipelines.
>
> The XLang effort has been focusing about getting more transforms working
> and solving ease of use issues for users so I don't believe what you
> describe has been done.
>
> On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:
>
>> /cc @Chamikara Jayalath <ch...@google.com> @Robert Bradshaw
>> <ro...@google.com>
>>
>> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org> wrote:
>>
>>> Has anyone ever tried hosting a long-running expansion service as a real
>>> "service", the intent being that users don't need to run it locally, and
>>> can instead connect to the shared one when expanding pipelines?
>>>
>>> Looking around the code I already see a few assumptions that it will
>>> only live for a short while (memory leaks, etc), as well as some pretty
>>> large security concerns (eg, the artifact retrieval service can be used to
>>> download any file on the host) curious if anyone has put any work into this
>>> already?
>>>
>>

Re: ExpansionService...as a service?

Posted by Luke Cwik <lc...@google.com>.
I believe that was one of the ideas behind an expansion service but to my
knowledge I don't know of anyone who has a long running expansion service.
I was thinking that the Apache Beam community at some point could host one
so that users don't need to spin up Java/Python/Go expansion services to
run a subset of XLang pipelines.

The XLang effort has been focusing about getting more transforms working
and solving ease of use issues for users so I don't believe what you
describe has been done.

On Wed, Oct 6, 2021 at 8:37 AM Ahmet Altay <al...@google.com> wrote:

> /cc @Chamikara Jayalath <ch...@google.com> @Robert Bradshaw
> <ro...@google.com>
>
> On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org> wrote:
>
>> Has anyone ever tried hosting a long-running expansion service as a real
>> "service", the intent being that users don't need to run it locally, and
>> can instead connect to the shared one when expanding pipelines?
>>
>> Looking around the code I already see a few assumptions that it will only
>> live for a short while (memory leaks, etc), as well as some pretty large
>> security concerns (eg, the artifact retrieval service can be used to
>> download any file on the host) curious if anyone has put any work into this
>> already?
>>
>

Re: ExpansionService...as a service?

Posted by Ahmet Altay <al...@google.com>.
/cc @Chamikara Jayalath <ch...@google.com> @Robert Bradshaw
<ro...@google.com>

On Wed, Oct 6, 2021 at 6:36 AM Steve Niemitz <sn...@apache.org> wrote:

> Has anyone ever tried hosting a long-running expansion service as a real
> "service", the intent being that users don't need to run it locally, and
> can instead connect to the shared one when expanding pipelines?
>
> Looking around the code I already see a few assumptions that it will only
> live for a short while (memory leaks, etc), as well as some pretty large
> security concerns (eg, the artifact retrieval service can be used to
> download any file on the host) curious if anyone has put any work into this
> already?
>