You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@beam.apache.org by Ross Vandegrift <ro...@cleardata.com> on 2020/10/09 19:10:01 UTC

beam rebuilds numpy on pipeline run

Hello,

Starting today, running a beam pipeline triggers a large reinstallation of
python modules.  For some reason, it forces full rebuilds from source - since
beam depends on numpy, this takes a long time.

There's nothing strange about my python setup.  I'm using python3.7 on debian
buster with the dataflow runner.  My venv is setup like this:
 python3 -m venv ~/.venvs/beam
 . ~/.venvs/beam/bin/activate
 python3 -m pip install --upgrade wheel
 python3 -m pip install --upgrade pip setuptools
 python3 -m pip install -r requirements.txt

My requirements.txt has:
  apache-beam[gcp]==2.23.0
  boto3==1.15.0

When it's building, `ps ax | grep python` shows me this:
  /home/ross/.venvs/beam/bin/python -m pip download --dest /tmp/dataflow-
requirements-cache -r requirements.txt --exists-action i --no-binary :all:

How do I prevent this?  It's far too slow to develop with, and our compliance
folks are likely to prohibit a tool that silently downloads & builds unknown
code.

Ross

Re: beam rebuilds numpy on pipeline run

Posted by Valentyn Tymofieiev <va...@google.com>.
Could you try removing  apache-beam[gcp]==2.23.0 from the  requirements.txt
file that you pass when you launch the pipeline? Beam will download itself
from pypi and stage itself anyway, there is no need to include it in the
requirements file.

Let me know if this helps.

On Fri, Oct 9, 2020 at 2:11 PM Brian Hulette <bh...@google.com> wrote:

> +Valentyn Tymofieiev <va...@google.com>
>
> This sounds like it's related to ARROW-8983 (pyarrow takes a long time to
> download after 0.16.0), discussed on the arrow dev list [2]. I'm not sure
> what would've triggered this to start happening for you today though.
>
> [1] https://issues.apache.org/jira/browse/ARROW-8983
> [2]
> https://lists.apache.org/thread.html/r9baa48a9d1517834c285f0f238f29fcf54405cb7cf1e681314239d7f%40%3Cdev.arrow.apache.org%3E
>
> On Fri, Oct 9, 2020 at 12:10 PM Ross Vandegrift <
> ross.vandegrift@cleardata.com> wrote:
>
>> Hello,
>>
>> Starting today, running a beam pipeline triggers a large reinstallation of
>> python modules.  For some reason, it forces full rebuilds from source -
>> since
>> beam depends on numpy, this takes a long time.
>>
>> There's nothing strange about my python setup.  I'm using python3.7 on
>> debian
>> buster with the dataflow runner.  My venv is setup like this:
>>  python3 -m venv ~/.venvs/beam
>>  . ~/.venvs/beam/bin/activate
>>  python3 -m pip install --upgrade wheel
>>  python3 -m pip install --upgrade pip setuptools
>>  python3 -m pip install -r requirements.txt
>>
>> My requirements.txt has:
>>   apache-beam[gcp]==2.23.0
>>   boto3==1.15.0
>>
>> When it's building, `ps ax | grep python` shows me this:
>>   /home/ross/.venvs/beam/bin/python -m pip download --dest /tmp/dataflow-
>> requirements-cache -r requirements.txt --exists-action i --no-binary :all:
>>
>> How do I prevent this?  It's far too slow to develop with, and our
>> compliance
>> folks are likely to prohibit a tool that silently downloads & builds
>> unknown
>> code.
>>
>> Ross
>>
>

Re: beam rebuilds numpy on pipeline run

Posted by Brian Hulette <bh...@google.com>.
+Valentyn Tymofieiev <va...@google.com>

This sounds like it's related to ARROW-8983 (pyarrow takes a long time to
download after 0.16.0), discussed on the arrow dev list [2]. I'm not sure
what would've triggered this to start happening for you today though.

[1] https://issues.apache.org/jira/browse/ARROW-8983
[2]
https://lists.apache.org/thread.html/r9baa48a9d1517834c285f0f238f29fcf54405cb7cf1e681314239d7f%40%3Cdev.arrow.apache.org%3E

On Fri, Oct 9, 2020 at 12:10 PM Ross Vandegrift <
ross.vandegrift@cleardata.com> wrote:

> Hello,
>
> Starting today, running a beam pipeline triggers a large reinstallation of
> python modules.  For some reason, it forces full rebuilds from source -
> since
> beam depends on numpy, this takes a long time.
>
> There's nothing strange about my python setup.  I'm using python3.7 on
> debian
> buster with the dataflow runner.  My venv is setup like this:
>  python3 -m venv ~/.venvs/beam
>  . ~/.venvs/beam/bin/activate
>  python3 -m pip install --upgrade wheel
>  python3 -m pip install --upgrade pip setuptools
>  python3 -m pip install -r requirements.txt
>
> My requirements.txt has:
>   apache-beam[gcp]==2.23.0
>   boto3==1.15.0
>
> When it's building, `ps ax | grep python` shows me this:
>   /home/ross/.venvs/beam/bin/python -m pip download --dest /tmp/dataflow-
> requirements-cache -r requirements.txt --exists-action i --no-binary :all:
>
> How do I prevent this?  It's far too slow to develop with, and our
> compliance
> folks are likely to prohibit a tool that silently downloads & builds
> unknown
> code.
>
> Ross
>

Re: beam rebuilds numpy on pipeline run

Posted by Ross Vandegrift <ro...@cleardata.com>.
On Fri, 2020-10-09 at 19:10 +0000, Ross Vandegrift wrote:
> Starting today, running a beam pipeline triggers a large reinstallation of
> python modules.  For some reason, it forces full rebuilds from source -
> since
> beam depends on numpy, this takes a long time.

I opened a support ticket with Google and got a workaround: move all of the
dependencies from requirements.txt to the setuptools invocation in setup.py. 
No more numpy rebuilds.

Ross