You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Alexey Romanenko <ar...@gmail.com> on 2020/05/15 15:46:42 UTC
Portable Runner performance optimisation
Hello,
I’m trying to optimize my pipeline runtime while using it with Portable Runner and I have some related questions.
This is a cross-language pipeline, written in Java SDK, and which executes some Python code through “External.of()” transform and my custom Python Expansion Service. I use Docker-based SDK Harness for Java and Python. In a primitive form the pipeline would look like this:
[Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python code with Python SDK) ] - > [MyTransform2 (Java SDK)]
While running this pipeline with Portable Spark Runner, I see that quite a lot of time we spend for artifacts staging (in our case, we have quite a lot of artifacts in real pipeline) and launching a Docker container for every Spark stage. So, my questions are the following:
1. Is there any internal Beam functionality to pre-stage or, at least cache, already staged artifacts? Since the same pipeline will be executed many times in a row, there is no reason to stage the same artifacts every run.
2. Is it possible to pre-run SDK Harness containers and reuse them for every Portable Runner pipeline? I could win quite a lot of time on this for more complicated pipelines.
Well, I guess I can find some workarounds for that but I wished to ask before that perhaps there is a better way to do that in Beam.
Regards,
Alexey
Re: Portable Runner performance optimisation
Posted by Robert Bradshaw <ro...@google.com>.
I don't think this is being worked on, but given that Java already supports
the LOOPBACK environment (which is a special case of EXTERNAL) it would
just be a matter of properly parsing the flags.
On Fri, May 15, 2020 at 9:52 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Thanks! It looks that this is exactly what I need, though mostly for Java
> SDK.
> Don't you know if anyone works on this Jira?
>
> On 15 May 2020, at 18:01, Kyle Weaver <kc...@google.com> wrote:
>
> > Yes, you can start docker containers before hand using the worker_pool
> option:
>
> However, it only works for Python. Java doesn't have it yet:
> https://issues.apache.org/jira/browse/BEAM-8137
>
> On Fri, May 15, 2020 at 12:00 PM Kyle Weaver <kc...@google.com> wrote:
>
>> > 2. Is it possible to pre-run SDK Harness containers and reuse them for
>> every Portable Runner pipeline? I could win quite a lot of time on this for
>> more complicated pipelines.
>>
>> Yes, you can start docker containers before hand using the worker_pool
>> option:
>>
>> docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or
>> some other port publishing
>>
>> and then in your pipeline options set:
>>
>> --environment_type=EXTERNAL --environment_config=localhost:50000
>>
>> On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I’m trying to optimize my pipeline runtime while using it with Portable
>>> Runner and I have some related questions.
>>>
>>> This is a cross-language pipeline, written in Java SDK, and which
>>> executes some Python code through “External.of()” transform and my custom
>>> Python Expansion Service. I use Docker-based SDK Harness for Java and
>>> Python. In a primitive form the pipeline would look like this:
>>>
>>>
>>> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python
>>> code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>>>
>>>
>>>
>>> While running this pipeline with Portable Spark Runner, I see that quite
>>> a lot of time we spend for artifacts staging (in our case, we have quite a
>>> lot of artifacts in real pipeline) and launching a Docker container for
>>> every Spark stage. So, my questions are the following:
>>>
>>> 1. Is there any internal Beam functionality to pre-stage or, at least
>>> cache, already staged artifacts? Since the same pipeline will be executed
>>> many times in a row, there is no reason to stage the same artifacts every
>>> run.
>>>
>>> 2. Is it possible to pre-run SDK Harness containers and reuse them for
>>> every Portable Runner pipeline? I could win quite a lot of time on this for
>>> more complicated pipelines.
>>>
>>>
>>>
>>> Well, I guess I can find some workarounds for that but I wished to ask
>>> before that perhaps there is a better way to do that in Beam.
>>>
>>>
>>> Regards,
>>> Alexey
>>
>>
>
Re: Portable Runner performance optimisation
Posted by Alexey Romanenko <ar...@gmail.com>.
Thanks! It looks that this is exactly what I need, though mostly for Java SDK.
Don't you know if anyone works on this Jira?
> On 15 May 2020, at 18:01, Kyle Weaver <kc...@google.com> wrote:
>
> > Yes, you can start docker containers before hand using the worker_pool option:
>
> However, it only works for Python. Java doesn't have it yet: https://issues.apache.org/jira/browse/BEAM-8137 <https://issues.apache.org/jira/browse/BEAM-8137>
> On Fri, May 15, 2020 at 12:00 PM Kyle Weaver <kcweaver@google.com <ma...@google.com>> wrote:
> > 2. Is it possible to pre-run SDK Harness containers and reuse them for every Portable Runner pipeline? I could win quite a lot of time on this for more complicated pipelines.
>
> Yes, you can start docker containers before hand using the worker_pool option:
>
> docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or some other port publishing
>
> and then in your pipeline options set:
>
> --environment_type=EXTERNAL --environment_config=localhost:50000
>
> On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <aromanenko.dev@gmail.com <ma...@gmail.com>> wrote:
> Hello,
>
> I’m trying to optimize my pipeline runtime while using it with Portable Runner and I have some related questions.
>
> This is a cross-language pipeline, written in Java SDK, and which executes some Python code through “External.of()” transform and my custom Python Expansion Service. I use Docker-based SDK Harness for Java and Python. In a primitive form the pipeline would look like this:
>
>
> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>
>
>
> While running this pipeline with Portable Spark Runner, I see that quite a lot of time we spend for artifacts staging (in our case, we have quite a lot of artifacts in real pipeline) and launching a Docker container for every Spark stage. So, my questions are the following:
>
> 1. Is there any internal Beam functionality to pre-stage or, at least cache, already staged artifacts? Since the same pipeline will be executed many times in a row, there is no reason to stage the same artifacts every run.
>
> 2. Is it possible to pre-run SDK Harness containers and reuse them for every Portable Runner pipeline? I could win quite a lot of time on this for more complicated pipelines.
>
>
>
> Well, I guess I can find some workarounds for that but I wished to ask before that perhaps there is a better way to do that in Beam.
>
>
> Regards,
> Alexey
Re: Portable Runner performance optimisation
Posted by Alexey Romanenko <ar...@gmail.com>.
> On 15 May 2020, at 18:15, Luke Cwik <lc...@google.com> wrote:
>
> 1. Is there any internal Beam functionality to pre-stage or, at least cache, already staged artifacts? Since the same pipeline will be executed many times in a row, there is no reason to stage the same artifacts every run.
>
>
> Part of artifact staging there is supposed to be deduplication but sometimes minor changes in the files like the jar gets recreated with the same contents but different times leads to a different hash breaking the deduplication.
Do you mean that it should work so by default or it should be explicitly enabled? Sounds like this is what I need.
>
> You can always embed your artifacts in your containers and try to make it so that you have zero artifacts to stage/retrieve.
Yes, this is what I called a “workaround” but in this case I would need to disable artifacts staging at all, I guess, by setting “--filesToStage” to empty directory?
>
>
> 2. Is it possible to pre-run SDK Harness containers and reuse them for every Portable Runner pipeline? I could win quite a lot of time on this for more complicated pipelines.
>
>
>
> Well, I guess I can find some workarounds for that but I wished to ask before that perhaps there is a better way to do that in Beam.
>
>
> Regards,
> Alexey
Re: Portable Runner performance optimisation
Posted by Kyle Weaver <kc...@google.com>.
Note also that a worker pool should only retrieve artifacts once:
https://github.com/apache/beam/pull/9398
On Fri, May 15, 2020 at 12:15 PM Luke Cwik <lc...@google.com> wrote:
>
>
> On Fri, May 15, 2020 at 9:01 AM Kyle Weaver <kc...@google.com> wrote:
>
>> > Yes, you can start docker containers before hand using the worker_pool
>> option:
>>
>> However, it only works for Python. Java doesn't have it yet:
>> https://issues.apache.org/jira/browse/BEAM-8137
>>
>> On Fri, May 15, 2020 at 12:00 PM Kyle Weaver <kc...@google.com> wrote:
>>
>>> > 2. Is it possible to pre-run SDK Harness containers and reuse them for
>>> every Portable Runner pipeline? I could win quite a lot of time on this for
>>> more complicated pipelines.
>>>
>>> Yes, you can start docker containers before hand using the worker_pool
>>> option:
>>>
>>> docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or
>>> some other port publishing
>>>
>>> and then in your pipeline options set:
>>>
>>> --environment_type=EXTERNAL --environment_config=localhost:50000
>>>
>>> On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <
>>> aromanenko.dev@gmail.com> wrote:
>>>
>>>> Hello,
>>>>
>>>> I’m trying to optimize my pipeline runtime while using it with Portable
>>>> Runner and I have some related questions.
>>>>
>>>> This is a cross-language pipeline, written in Java SDK, and which
>>>> executes some Python code through “External.of()” transform and my custom
>>>> Python Expansion Service. I use Docker-based SDK Harness for Java and
>>>> Python. In a primitive form the pipeline would look like this:
>>>>
>>>>
>>>> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python
>>>> code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>>>>
>>>>
>>>>
>>>> While running this pipeline with Portable Spark Runner, I see that
>>>> quite a lot of time we spend for artifacts staging (in our case, we have
>>>> quite a lot of artifacts in real pipeline) and launching a Docker container
>>>> for every Spark stage. So, my questions are the following:
>>>>
>>>> 1. Is there any internal Beam functionality to pre-stage or, at least
>>>> cache, already staged artifacts? Since the same pipeline will be executed
>>>> many times in a row, there is no reason to stage the same artifacts every
>>>> run.
>>>>
>>>>
> Part of artifact staging there is supposed to be deduplication but
> sometimes minor changes in the files like the jar gets recreated with the
> same contents but different times leads to a different hash breaking the
> deduplication.
>
> You can always embed your artifacts in your containers and try to make it
> so that you have zero artifacts to stage/retrieve.
>
>
>
>> 2. Is it possible to pre-run SDK Harness containers and reuse them for
>>>> every Portable Runner pipeline? I could win quite a lot of time on this for
>>>> more complicated pipelines.
>>>>
>>>>
>>>>
>>>> Well, I guess I can find some workarounds for that but I wished to ask
>>>> before that perhaps there is a better way to do that in Beam.
>>>>
>>>>
>>>> Regards,
>>>> Alexey
>>>
>>>
Re: Portable Runner performance optimisation
Posted by Luke Cwik <lc...@google.com>.
On Fri, May 15, 2020 at 9:01 AM Kyle Weaver <kc...@google.com> wrote:
> > Yes, you can start docker containers before hand using the worker_pool
> option:
>
> However, it only works for Python. Java doesn't have it yet:
> https://issues.apache.org/jira/browse/BEAM-8137
>
> On Fri, May 15, 2020 at 12:00 PM Kyle Weaver <kc...@google.com> wrote:
>
>> > 2. Is it possible to pre-run SDK Harness containers and reuse them for
>> every Portable Runner pipeline? I could win quite a lot of time on this for
>> more complicated pipelines.
>>
>> Yes, you can start docker containers before hand using the worker_pool
>> option:
>>
>> docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or
>> some other port publishing
>>
>> and then in your pipeline options set:
>>
>> --environment_type=EXTERNAL --environment_config=localhost:50000
>>
>> On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <
>> aromanenko.dev@gmail.com> wrote:
>>
>>> Hello,
>>>
>>> I’m trying to optimize my pipeline runtime while using it with Portable
>>> Runner and I have some related questions.
>>>
>>> This is a cross-language pipeline, written in Java SDK, and which
>>> executes some Python code through “External.of()” transform and my custom
>>> Python Expansion Service. I use Docker-based SDK Harness for Java and
>>> Python. In a primitive form the pipeline would look like this:
>>>
>>>
>>> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python
>>> code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>>>
>>>
>>>
>>> While running this pipeline with Portable Spark Runner, I see that quite
>>> a lot of time we spend for artifacts staging (in our case, we have quite a
>>> lot of artifacts in real pipeline) and launching a Docker container for
>>> every Spark stage. So, my questions are the following:
>>>
>>> 1. Is there any internal Beam functionality to pre-stage or, at least
>>> cache, already staged artifacts? Since the same pipeline will be executed
>>> many times in a row, there is no reason to stage the same artifacts every
>>> run.
>>>
>>>
Part of artifact staging there is supposed to be deduplication but
sometimes minor changes in the files like the jar gets recreated with the
same contents but different times leads to a different hash breaking the
deduplication.
You can always embed your artifacts in your containers and try to make it
so that you have zero artifacts to stage/retrieve.
> 2. Is it possible to pre-run SDK Harness containers and reuse them for
>>> every Portable Runner pipeline? I could win quite a lot of time on this for
>>> more complicated pipelines.
>>>
>>>
>>>
>>> Well, I guess I can find some workarounds for that but I wished to ask
>>> before that perhaps there is a better way to do that in Beam.
>>>
>>>
>>> Regards,
>>> Alexey
>>
>>
Re: Portable Runner performance optimisation
Posted by Kyle Weaver <kc...@google.com>.
> Yes, you can start docker containers before hand using the worker_pool
option:
However, it only works for Python. Java doesn't have it yet:
https://issues.apache.org/jira/browse/BEAM-8137
On Fri, May 15, 2020 at 12:00 PM Kyle Weaver <kc...@google.com> wrote:
> > 2. Is it possible to pre-run SDK Harness containers and reuse them for
> every Portable Runner pipeline? I could win quite a lot of time on this for
> more complicated pipelines.
>
> Yes, you can start docker containers before hand using the worker_pool
> option:
>
> docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or some
> other port publishing
>
> and then in your pipeline options set:
>
> --environment_type=EXTERNAL --environment_config=localhost:50000
>
> On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <
> aromanenko.dev@gmail.com> wrote:
>
>> Hello,
>>
>> I’m trying to optimize my pipeline runtime while using it with Portable
>> Runner and I have some related questions.
>>
>> This is a cross-language pipeline, written in Java SDK, and which
>> executes some Python code through “External.of()” transform and my custom
>> Python Expansion Service. I use Docker-based SDK Harness for Java and
>> Python. In a primitive form the pipeline would look like this:
>>
>>
>> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python
>> code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>>
>>
>>
>> While running this pipeline with Portable Spark Runner, I see that quite
>> a lot of time we spend for artifacts staging (in our case, we have quite a
>> lot of artifacts in real pipeline) and launching a Docker container for
>> every Spark stage. So, my questions are the following:
>>
>> 1. Is there any internal Beam functionality to pre-stage or, at least
>> cache, already staged artifacts? Since the same pipeline will be executed
>> many times in a row, there is no reason to stage the same artifacts every
>> run.
>>
>> 2. Is it possible to pre-run SDK Harness containers and reuse them for
>> every Portable Runner pipeline? I could win quite a lot of time on this for
>> more complicated pipelines.
>>
>>
>>
>> Well, I guess I can find some workarounds for that but I wished to ask
>> before that perhaps there is a better way to do that in Beam.
>>
>>
>> Regards,
>> Alexey
>
>
Re: Portable Runner performance optimisation
Posted by Kyle Weaver <kc...@google.com>.
> 2. Is it possible to pre-run SDK Harness containers and reuse them for
every Portable Runner pipeline? I could win quite a lot of time on this for
more complicated pipelines.
Yes, you can start docker containers before hand using the worker_pool
option:
docker run -p=50000:50000 apachebeam/python3.7_sdk --worker_pool # or some
other port publishing
and then in your pipeline options set:
--environment_type=EXTERNAL --environment_config=localhost:50000
On Fri, May 15, 2020 at 11:47 AM Alexey Romanenko <ar...@gmail.com>
wrote:
> Hello,
>
> I’m trying to optimize my pipeline runtime while using it with Portable
> Runner and I have some related questions.
>
> This is a cross-language pipeline, written in Java SDK, and which executes
> some Python code through “External.of()” transform and my custom Python
> Expansion Service. I use Docker-based SDK Harness for Java and Python. In a
> primitive form the pipeline would look like this:
>
>
> [Source (Java)] -> [MyTransform1 (Java)] -> [External (Execute Python
> code with Python SDK) ] - > [MyTransform2 (Java SDK)]
>
>
>
> While running this pipeline with Portable Spark Runner, I see that quite a
> lot of time we spend for artifacts staging (in our case, we have quite a
> lot of artifacts in real pipeline) and launching a Docker container for
> every Spark stage. So, my questions are the following:
>
> 1. Is there any internal Beam functionality to pre-stage or, at least
> cache, already staged artifacts? Since the same pipeline will be executed
> many times in a row, there is no reason to stage the same artifacts every
> run.
>
> 2. Is it possible to pre-run SDK Harness containers and reuse them for
> every Portable Runner pipeline? I could win quite a lot of time on this for
> more complicated pipelines.
>
>
>
> Well, I guess I can find some workarounds for that but I wished to ask
> before that perhaps there is a better way to do that in Beam.
>
>
> Regards,
> Alexey