You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Kenneth Knowles <ke...@apache.org> on 2019/01/16 04:22:22 UTC
gradle clean causes long-running python installs
A global `./gradlew clean` runs various `setupVirtualEnv` tasks that invoke
things such as `setup.py bdist_wheel for grpcio-tools`. Overall it took 4
minutes. Is this intended?
Kenn
Re: gradle clean causes long-running python installs
Posted by Udi Meiri <eh...@google.com>.
The dependency of clean on setupVirtualenv has been removed.
I use gradle for daily development on python, but only to run the "lint"
task.
There are a lot of python build artifacts that "clean" doesn't delete.
On Wed, Feb 20, 2019 at 1:17 AM Michael Luckey <ad...@gmail.com> wrote:
> Thats said, removing this dependency on setupVirtualEnv is definitely
> a good idea.
>
> On Wed, Feb 20, 2019 at 10:14 AM Michael Luckey <ad...@gmail.com>
> wrote:
>
>> Is anyone using gradle for daily development of python parts? Or is this
>> just used as integration for Jenkins build? In case of the latter, clean
>> isn't called anyway, right?
>>
>> If the former, its probably worth to put more effort into python setup
>> anyway, e.g. declaring proper input/outputs... If done correctly (if thats
>> possible at all) clean will work out of the box anyway.
>>
>> On Wed, Feb 20, 2019 at 1:01 AM Udi Meiri <eh...@google.com> wrote:
>>
>>> I think I can solve this issue by removing the dependency and adding a
>>> check to see if the virtualenv was created.
>>> Otherwise, there shouldn't be anything to cleanup anyway.
>>>
>>> On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <ry...@runsascoded.com>
>>> wrote:
>>>
>>>> Thanks Michael, your assessment was correct. I needed python3.5 on my
>>>> $PATH.
>>>>
>>>> For completeness, I needed this
>>>> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib>
>>>> to get pyenv to install python 3.5.6 on macOS:
>>>>
>>>> ```
>>>> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6
>>>> ```
>>>>
>>>> `./gradlew clean` worked after that.
>>>>
>>>>
>>>> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <ad...@gmail.com>
>>>> wrote:
>>>>
>>>>> As far as I understand, the build got bound to 3.5 [1].
>>>>>
>>>>> Could it be that you do not have python3.5 on your path? e.g. try python3.5
>>>>> --version
>>>>>
>>>>> If that is missing, you will not be able to run any py3 task, I
>>>>> guess...
>>>>>
>>>>> So, to get out of this state, you have to get python3.5 command
>>>>> working.
>>>>>
>>>>> At least on my machine, './gradlew clean' is working iff python3.5 is
>>>>> on my path.
>>>>>
>>>>> michel
>>>>>
>>>>> [1]
>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
>>>>>
>>>>>
>>>>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com>
>>>>> wrote:
>>>>>
>>>>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on
>>>>>> two py3 setupVirtualEnv tasks,
>>>>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
>>>>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's
>>>>>> full output
>>>>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>
>>>>>> .
>>>>>>
>>>>>> Any tips how to get out of this state, or what is causing it?
>>>>>>
>>>>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Maybe add that case to
>>>>>>> https://issues.apache.org/jira/browse/BEAM-6459.
>>>>>>>
>>>>>>> Kenn
>>>>>>>
>>>>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>>
>>>>>>>> Probably related, a top level ./gradlew clean fails with the
>>>>>>>> following:
>>>>>>>>
>>>>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>>>>>>> The path python3.5 (from --python=python3.5) does not exist
>>>>>>>>
>>>>>>>> Can we just limit clean to do cleanup?!
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <
>>>>>>>> robertwb@google.com> wrote:
>>>>>>>>
>>>>>>>>> Just some background, grpcio-tools is what's used to do the proto
>>>>>>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>>>>>>> provide very many wheels, so we want to install it once, not every
>>>>>>>>> time. (It's also used in more than just tests; one needs it every
>>>>>>>>> time
>>>>>>>>> the .proto files change.)
>>>>>>>>>
>>>>>>>>> That being said, we could probably do a much cheaper clean.
>>>>>>>>>
>>>>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >
>>>>>>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>>>>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>>>>>>> >
>>>>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>
>>>>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>>>>>>> without installing gcpio-tools, etc?
>>>>>>>>> >>
>>>>>>>>> >> Kenn
>>>>>>>>> >>
>>>>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com>
>>>>>>>>> wrote:
>>>>>>>>> >>>
>>>>>>>>> >>> setup.py has requirements like setuptools, which are installed
>>>>>>>>> in the virtual environment.
>>>>>>>>> >>> So even running the clean command requires the virtualenv to
>>>>>>>>> be set up.
>>>>>>>>> >>>
>>>>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython
>>>>>>>>> if setupVirtualenv has not been run. (perhaps by checking for the existence
>>>>>>>>> of its output directory)
>>>>>>>>> >>>
>>>>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <
>>>>>>>>> klk@google.com> wrote:
>>>>>>>>> >>>>
>>>>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to
>>>>>>>>> record the conclusion. Doesn't require Beam knowledge so I labeled
>>>>>>>>> "starter".
>>>>>>>>> >>>>
>>>>>>>>> >>>> Kenn
>>>>>>>>> >>>>
>>>>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>>>>>>> adude3141@gmail.com> wrote:
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> This seems to be on purpose [1]
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean.
>>>>>>>>> We probably should work around that.
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> [1]
>>>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>>>>>> >>>>>
>>>>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>>>>>>> owenzhang1990@gmail.com> wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean`
>>>>>>>>> fails due to failure of `setupVirtualEnv` tasks.
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Manu Zhang
>>>>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>>>>>>> kenn@apache.org>, wrote:
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv`
>>>>>>>>> tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`.
>>>>>>>>> Overall it took 4 minutes. Is this intended?
>>>>>>>>> >>>>>>
>>>>>>>>> >>>>>> Kenn
>>>>>>>>>
>>>>>>>>
Re: gradle clean causes long-running python installs
Posted by Michael Luckey <ad...@gmail.com>.
Thats said, removing this dependency on setupVirtualEnv is definitely
a good idea.
On Wed, Feb 20, 2019 at 10:14 AM Michael Luckey <ad...@gmail.com> wrote:
> Is anyone using gradle for daily development of python parts? Or is this
> just used as integration for Jenkins build? In case of the latter, clean
> isn't called anyway, right?
>
> If the former, its probably worth to put more effort into python setup
> anyway, e.g. declaring proper input/outputs... If done correctly (if thats
> possible at all) clean will work out of the box anyway.
>
> On Wed, Feb 20, 2019 at 1:01 AM Udi Meiri <eh...@google.com> wrote:
>
>> I think I can solve this issue by removing the dependency and adding a
>> check to see if the virtualenv was created.
>> Otherwise, there shouldn't be anything to cleanup anyway.
>>
>> On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <ry...@runsascoded.com>
>> wrote:
>>
>>> Thanks Michael, your assessment was correct. I needed python3.5 on my
>>> $PATH.
>>>
>>> For completeness, I needed this
>>> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib>
>>> to get pyenv to install python 3.5.6 on macOS:
>>>
>>> ```
>>> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6
>>> ```
>>>
>>> `./gradlew clean` worked after that.
>>>
>>>
>>> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <ad...@gmail.com>
>>> wrote:
>>>
>>>> As far as I understand, the build got bound to 3.5 [1].
>>>>
>>>> Could it be that you do not have python3.5 on your path? e.g. try python3.5
>>>> --version
>>>>
>>>> If that is missing, you will not be able to run any py3 task, I guess...
>>>>
>>>> So, to get out of this state, you have to get python3.5 command working.
>>>>
>>>> At least on my machine, './gradlew clean' is working iff python3.5 is
>>>> on my path.
>>>>
>>>> michel
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
>>>>
>>>>
>>>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com>
>>>> wrote:
>>>>
>>>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on
>>>>> two py3 setupVirtualEnv tasks,
>>>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
>>>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's
>>>>> full output
>>>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>
>>>>> .
>>>>>
>>>>> Any tips how to get out of this state, or what is causing it?
>>>>>
>>>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>>
>>>>>> Maybe add that case to
>>>>>> https://issues.apache.org/jira/browse/BEAM-6459.
>>>>>>
>>>>>> Kenn
>>>>>>
>>>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>>>>>
>>>>>>> Probably related, a top level ./gradlew clean fails with the
>>>>>>> following:
>>>>>>>
>>>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>>>>>> The path python3.5 (from --python=python3.5) does not exist
>>>>>>>
>>>>>>> Can we just limit clean to do cleanup?!
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Just some background, grpcio-tools is what's used to do the proto
>>>>>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>>>>>> provide very many wheels, so we want to install it once, not every
>>>>>>>> time. (It's also used in more than just tests; one needs it every
>>>>>>>> time
>>>>>>>> the .proto files change.)
>>>>>>>>
>>>>>>>> That being said, we could probably do a much cheaper clean.
>>>>>>>>
>>>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>>> >
>>>>>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>>>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>>>>>> >
>>>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>
>>>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>>>>>> without installing gcpio-tools, etc?
>>>>>>>> >>
>>>>>>>> >> Kenn
>>>>>>>> >>
>>>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>
>>>>>>>> >>> setup.py has requirements like setuptools, which are installed
>>>>>>>> in the virtual environment.
>>>>>>>> >>> So even running the clean command requires the virtualenv to be
>>>>>>>> set up.
>>>>>>>> >>>
>>>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython
>>>>>>>> if setupVirtualenv has not been run. (perhaps by checking for the existence
>>>>>>>> of its output directory)
>>>>>>>> >>>
>>>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>>>>>>> wrote:
>>>>>>>> >>>>
>>>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to
>>>>>>>> record the conclusion. Doesn't require Beam knowledge so I labeled
>>>>>>>> "starter".
>>>>>>>> >>>>
>>>>>>>> >>>> Kenn
>>>>>>>> >>>>
>>>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>>>>>> adude3141@gmail.com> wrote:
>>>>>>>> >>>>>
>>>>>>>> >>>>> This seems to be on purpose [1]
>>>>>>>> >>>>>
>>>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean.
>>>>>>>> We probably should work around that.
>>>>>>>> >>>>>
>>>>>>>> >>>>> [1]
>>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>>>>> >>>>>
>>>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>>>>>> owenzhang1990@gmail.com> wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean`
>>>>>>>> fails due to failure of `setupVirtualEnv` tasks.
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Manu Zhang
>>>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>>>>>> kenn@apache.org>, wrote:
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv`
>>>>>>>> tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`.
>>>>>>>> Overall it took 4 minutes. Is this intended?
>>>>>>>> >>>>>>
>>>>>>>> >>>>>> Kenn
>>>>>>>>
>>>>>>>
Re: gradle clean causes long-running python installs
Posted by Michael Luckey <ad...@gmail.com>.
Is anyone using gradle for daily development of python parts? Or is this
just used as integration for Jenkins build? In case of the latter, clean
isn't called anyway, right?
If the former, its probably worth to put more effort into python setup
anyway, e.g. declaring proper input/outputs... If done correctly (if thats
possible at all) clean will work out of the box anyway.
On Wed, Feb 20, 2019 at 1:01 AM Udi Meiri <eh...@google.com> wrote:
> I think I can solve this issue by removing the dependency and adding a
> check to see if the virtualenv was created.
> Otherwise, there shouldn't be anything to cleanup anyway.
>
> On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <ry...@runsascoded.com>
> wrote:
>
>> Thanks Michael, your assessment was correct. I needed python3.5 on my
>> $PATH.
>>
>> For completeness, I needed this
>> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib>
>> to get pyenv to install python 3.5.6 on macOS:
>>
>> ```
>> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6
>> ```
>>
>> `./gradlew clean` worked after that.
>>
>>
>> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <ad...@gmail.com>
>> wrote:
>>
>>> As far as I understand, the build got bound to 3.5 [1].
>>>
>>> Could it be that you do not have python3.5 on your path? e.g. try python3.5
>>> --version
>>>
>>> If that is missing, you will not be able to run any py3 task, I guess...
>>>
>>> So, to get out of this state, you have to get python3.5 command working.
>>>
>>> At least on my machine, './gradlew clean' is working iff python3.5 is on
>>> my path.
>>>
>>> michel
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
>>>
>>>
>>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com>
>>> wrote:
>>>
>>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on
>>>> two py3 setupVirtualEnv tasks,
>>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
>>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's
>>>> full output
>>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>
>>>> .
>>>>
>>>> Any tips how to get out of this state, or what is causing it?
>>>>
>>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>>> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459
>>>>> .
>>>>>
>>>>> Kenn
>>>>>
>>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>>>>
>>>>>> Probably related, a top level ./gradlew clean fails with the
>>>>>> following:
>>>>>>
>>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>>>>> The path python3.5 (from --python=python3.5) does not exist
>>>>>>
>>>>>> Can we just limit clean to do cleanup?!
>>>>>>
>>>>>>
>>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>>>>>> wrote:
>>>>>>
>>>>>>> Just some background, grpcio-tools is what's used to do the proto
>>>>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>>>>> provide very many wheels, so we want to install it once, not every
>>>>>>> time. (It's also used in more than just tests; one needs it every
>>>>>>> time
>>>>>>> the .proto files change.)
>>>>>>>
>>>>>>> That being said, we could probably do a much cheaper clean.
>>>>>>>
>>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>>>>>> >
>>>>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>>>>> >
>>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>> >>
>>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>>>>> without installing gcpio-tools, etc?
>>>>>>> >>
>>>>>>> >> Kenn
>>>>>>> >>
>>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com>
>>>>>>> wrote:
>>>>>>> >>>
>>>>>>> >>> setup.py has requirements like setuptools, which are installed
>>>>>>> in the virtual environment.
>>>>>>> >>> So even running the clean command requires the virtualenv to be
>>>>>>> set up.
>>>>>>> >>>
>>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>>>>>>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>>>>>>> its output directory)
>>>>>>> >>>
>>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>>>>>> wrote:
>>>>>>> >>>>
>>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to
>>>>>>> record the conclusion. Doesn't require Beam knowledge so I labeled
>>>>>>> "starter".
>>>>>>> >>>>
>>>>>>> >>>> Kenn
>>>>>>> >>>>
>>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>>>>> adude3141@gmail.com> wrote:
>>>>>>> >>>>>
>>>>>>> >>>>> This seems to be on purpose [1]
>>>>>>> >>>>>
>>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>>>>>>> probably should work around that.
>>>>>>> >>>>>
>>>>>>> >>>>> [1]
>>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>>>> >>>>>
>>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>>>>> owenzhang1990@gmail.com> wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean`
>>>>>>> fails due to failure of `setupVirtualEnv` tasks.
>>>>>>> >>>>>>
>>>>>>> >>>>>> Manu Zhang
>>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>>>>> kenn@apache.org>, wrote:
>>>>>>> >>>>>>
>>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv`
>>>>>>> tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`.
>>>>>>> Overall it took 4 minutes. Is this intended?
>>>>>>> >>>>>>
>>>>>>> >>>>>> Kenn
>>>>>>>
>>>>>>
Re: gradle clean causes long-running python installs
Posted by Udi Meiri <eh...@google.com>.
I think I can solve this issue by removing the dependency and adding a
check to see if the virtualenv was created.
Otherwise, there shouldn't be anything to cleanup anyway.
On Sat, Feb 16, 2019 at 8:04 PM Ryan Williams <ry...@runsascoded.com> wrote:
> Thanks Michael, your assessment was correct. I needed python3.5 on my
> $PATH.
>
> For completeness, I needed this
> <https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib>
> to get pyenv to install python 3.5.6 on macOS:
>
> ```
> CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6
> ```
>
> `./gradlew clean` worked after that.
>
>
> On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <ad...@gmail.com>
> wrote:
>
>> As far as I understand, the build got bound to 3.5 [1].
>>
>> Could it be that you do not have python3.5 on your path? e.g. try python3.5
>> --version
>>
>> If that is missing, you will not be able to run any py3 task, I guess...
>>
>> So, to get out of this state, you have to get python3.5 command working.
>>
>> At least on my machine, './gradlew clean' is working iff python3.5 is on
>> my path.
>>
>> michel
>>
>> [1]
>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
>>
>>
>> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com>
>> wrote:
>>
>>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on
>>> two py3 setupVirtualEnv tasks,
>>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
>>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's full
>>> output
>>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>
>>> .
>>>
>>> Any tips how to get out of this state, or what is causing it?
>>>
>>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>>
>>>> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459.
>>>>
>>>> Kenn
>>>>
>>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>>>
>>>>> Probably related, a top level ./gradlew clean fails with the following:
>>>>>
>>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>>>> The path python3.5 (from --python=python3.5) does not exist
>>>>>
>>>>> Can we just limit clean to do cleanup?!
>>>>>
>>>>>
>>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>>>>> wrote:
>>>>>
>>>>>> Just some background, grpcio-tools is what's used to do the proto
>>>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>>>> provide very many wheels, so we want to install it once, not every
>>>>>> time. (It's also used in more than just tests; one needs it every time
>>>>>> the .proto files change.)
>>>>>>
>>>>>> That being said, we could probably do a much cheaper clean.
>>>>>>
>>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>>>>> >
>>>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>>>> >
>>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>> >>
>>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>>>> without installing gcpio-tools, etc?
>>>>>> >>
>>>>>> >> Kenn
>>>>>> >>
>>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com>
>>>>>> wrote:
>>>>>> >>>
>>>>>> >>> setup.py has requirements like setuptools, which are installed in
>>>>>> the virtual environment.
>>>>>> >>> So even running the clean command requires the virtualenv to be
>>>>>> set up.
>>>>>> >>>
>>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>>>>>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>>>>>> its output directory)
>>>>>> >>>
>>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>>>>> wrote:
>>>>>> >>>>
>>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record
>>>>>> the conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>>>> >>>>
>>>>>> >>>> Kenn
>>>>>> >>>>
>>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>>>> adude3141@gmail.com> wrote:
>>>>>> >>>>>
>>>>>> >>>>> This seems to be on purpose [1]
>>>>>> >>>>>
>>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>>>>>> probably should work around that.
>>>>>> >>>>>
>>>>>> >>>>> [1]
>>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>>> >>>>>
>>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>>>> owenzhang1990@gmail.com> wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean`
>>>>>> fails due to failure of `setupVirtualEnv` tasks.
>>>>>> >>>>>>
>>>>>> >>>>>> Manu Zhang
>>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>>>> kenn@apache.org>, wrote:
>>>>>> >>>>>>
>>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv`
>>>>>> tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`.
>>>>>> Overall it took 4 minutes. Is this intended?
>>>>>> >>>>>>
>>>>>> >>>>>> Kenn
>>>>>>
>>>>>
Re: gradle clean causes long-running python installs
Posted by Ryan Williams <ry...@runsascoded.com>.
Thanks Michael, your assessment was correct. I needed python3.5 on my $PATH.
For completeness, I needed this
<https://github.com/pyenv/pyenv/wiki/Common-build-problems#build-failed-error-the-python-zlib-extension-was-not-compiled-missing-the-zlib>
to get pyenv to install python 3.5.6 on macOS:
```
CPPFLAGS="-I$(brew --prefix zlib)/include" pyenv install -v 3.5.6
```
`./gradlew clean` worked after that.
On Sat, Feb 16, 2019 at 7:28 PM Michael Luckey <ad...@gmail.com> wrote:
> As far as I understand, the build got bound to 3.5 [1].
>
> Could it be that you do not have python3.5 on your path? e.g. try python3.5
> --version
>
> If that is missing, you will not be able to run any py3 task, I guess...
>
> So, to get out of this state, you have to get python3.5 command working.
>
> At least on my machine, './gradlew clean' is working iff python3.5 is on
> my path.
>
> michel
>
> [1]
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
>
>
> On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com>
> wrote:
>
>> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on two
>> py3 setupVirtualEnv tasks,
>> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
>> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's full
>> output
>> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>.
>>
>> Any tips how to get out of this state, or what is causing it?
>>
>> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459.
>>>
>>> Kenn
>>>
>>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>>
>>>> Probably related, a top level ./gradlew clean fails with the following:
>>>>
>>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>>> The path python3.5 (from --python=python3.5) does not exist
>>>>
>>>> Can we just limit clean to do cleanup?!
>>>>
>>>>
>>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>>>> wrote:
>>>>
>>>>> Just some background, grpcio-tools is what's used to do the proto
>>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>>> provide very many wheels, so we want to install it once, not every
>>>>> time. (It's also used in more than just tests; one needs it every time
>>>>> the .proto files change.)
>>>>>
>>>>> That being said, we could probably do a much cheaper clean.
>>>>>
>>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>>>> >
>>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>>> >
>>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>> >>
>>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>>> without installing gcpio-tools, etc?
>>>>> >>
>>>>> >> Kenn
>>>>> >>
>>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com>
>>>>> wrote:
>>>>> >>>
>>>>> >>> setup.py has requirements like setuptools, which are installed in
>>>>> the virtual environment.
>>>>> >>> So even running the clean command requires the virtualenv to be
>>>>> set up.
>>>>> >>>
>>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>>>>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>>>>> its output directory)
>>>>> >>>
>>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>>>> wrote:
>>>>> >>>>
>>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record
>>>>> the conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>>> >>>>
>>>>> >>>> Kenn
>>>>> >>>>
>>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>>> adude3141@gmail.com> wrote:
>>>>> >>>>>
>>>>> >>>>> This seems to be on purpose [1]
>>>>> >>>>>
>>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>>>>> probably should work around that.
>>>>> >>>>>
>>>>> >>>>> [1]
>>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>> >>>>>
>>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>>> owenzhang1990@gmail.com> wrote:
>>>>> >>>>>>
>>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean`
>>>>> fails due to failure of `setupVirtualEnv` tasks.
>>>>> >>>>>>
>>>>> >>>>>> Manu Zhang
>>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>>> kenn@apache.org>, wrote:
>>>>> >>>>>>
>>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks
>>>>> that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall
>>>>> it took 4 minutes. Is this intended?
>>>>> >>>>>>
>>>>> >>>>>> Kenn
>>>>>
>>>>
Re: gradle clean causes long-running python installs
Posted by Michael Luckey <ad...@gmail.com>.
As far as I understand, the build got bound to 3.5 [1].
Could it be that you do not have python3.5 on your path? e.g. try python3.5
--version
If that is missing, you will not be able to run any py3 task, I guess...
So, to get out of this state, you have to get python3.5 command working.
At least on my machine, './gradlew clean' is working iff python3.5 is on my
path.
michel
[1]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1591
On Sat, Feb 16, 2019 at 9:54 PM Ryan Williams <ry...@runsascoded.com> wrote:
> I'm seeing the same thing as Thomas above: `./gradlew clean` fails on two
> py3 setupVirtualEnv tasks,
> :beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
> :beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's full
> output
> <https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>.
>
> Any tips how to get out of this state, or what is causing it?
>
> On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
>
>> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459.
>>
>> Kenn
>>
>> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>>
>>> Probably related, a top level ./gradlew clean fails with the following:
>>>
>>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>>> The path python3.5 (from --python=python3.5) does not exist
>>>
>>> Can we just limit clean to do cleanup?!
>>>
>>>
>>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>>> wrote:
>>>
>>>> Just some background, grpcio-tools is what's used to do the proto
>>>> generation. Unfortunately it's expensive to compile and doesn't
>>>> provide very many wheels, so we want to install it once, not every
>>>> time. (It's also used in more than just tests; one needs it every time
>>>> the .proto files change.)
>>>>
>>>> That being said, we could probably do a much cheaper clean.
>>>>
>>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>>> >
>>>> > grpcio-tools could probably be moved under the "test" tag in
>>>> setup.py. Not sure why it has to be specified in gradle configs.
>>>> >
>>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>>> wrote:
>>>> >>
>>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean`
>>>> without installing gcpio-tools, etc?
>>>> >>
>>>> >> Kenn
>>>> >>
>>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>>>> >>>
>>>> >>> setup.py has requirements like setuptools, which are installed in
>>>> the virtual environment.
>>>> >>> So even running the clean command requires the virtualenv to be set
>>>> up.
>>>> >>>
>>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>>>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>>>> its output directory)
>>>> >>>
>>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>>> wrote:
>>>> >>>>
>>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record
>>>> the conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>> >>>>
>>>> >>>> Kenn
>>>> >>>>
>>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>>> adude3141@gmail.com> wrote:
>>>> >>>>>
>>>> >>>>> This seems to be on purpose [1]
>>>> >>>>>
>>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>>>> probably should work around that.
>>>> >>>>>
>>>> >>>>> [1]
>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>> >>>>>
>>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>>> owenzhang1990@gmail.com> wrote:
>>>> >>>>>>
>>>> >>>>>> I have the same question. Sometimes even `./gradlew clean` fails
>>>> due to failure of `setupVirtualEnv` tasks.
>>>> >>>>>>
>>>> >>>>>> Manu Zhang
>>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <
>>>> kenn@apache.org>, wrote:
>>>> >>>>>>
>>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks
>>>> that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall
>>>> it took 4 minutes. Is this intended?
>>>> >>>>>>
>>>> >>>>>> Kenn
>>>>
>>>
Re: gradle clean causes long-running python installs
Posted by Ryan Williams <ry...@runsascoded.com>.
I'm seeing the same thing as Thomas above: `./gradlew clean` fails on two
py3 setupVirtualEnv tasks,
:beam-sdks-python-test-suites-direct-py3:setupVirtualenv and
:beam-sdks-python-test-suites-dataflow-py3:setupVirtualenv. Here's full
output
<https://gist.github.com/ryan-williams/402e20131d23905163de3a4e2b178f39>.
Any tips how to get out of this state, or what is causing it?
On Fri, Feb 8, 2019 at 7:17 PM Kenneth Knowles <kl...@google.com> wrote:
> Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459.
>
> Kenn
>
> On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
>
>> Probably related, a top level ./gradlew clean fails with the following:
>>
>> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
>> The path python3.5 (from --python=python3.5) does not exist
>>
>> Can we just limit clean to do cleanup?!
>>
>>
>> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
>> wrote:
>>
>>> Just some background, grpcio-tools is what's used to do the proto
>>> generation. Unfortunately it's expensive to compile and doesn't
>>> provide very many wheels, so we want to install it once, not every
>>> time. (It's also used in more than just tests; one needs it every time
>>> the .proto files change.)
>>>
>>> That being said, we could probably do a much cheaper clean.
>>>
>>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>>> >
>>> > grpcio-tools could probably be moved under the "test" tag in setup.py.
>>> Not sure why it has to be specified in gradle configs.
>>> >
>>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>>> wrote:
>>> >>
>>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean` without
>>> installing gcpio-tools, etc?
>>> >>
>>> >> Kenn
>>> >>
>>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>>> >>>
>>> >>> setup.py has requirements like setuptools, which are installed in
>>> the virtual environment.
>>> >>> So even running the clean command requires the virtualenv to be set
>>> up.
>>> >>>
>>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>>> its output directory)
>>> >>>
>>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>>> wrote:
>>> >>>>
>>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record
>>> the conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>> >>>>
>>> >>>> Kenn
>>> >>>>
>>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <
>>> adude3141@gmail.com> wrote:
>>> >>>>>
>>> >>>>> This seems to be on purpose [1]
>>> >>>>>
>>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>>> probably should work around that.
>>> >>>>>
>>> >>>>> [1]
>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>> >>>>>
>>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <
>>> owenzhang1990@gmail.com> wrote:
>>> >>>>>>
>>> >>>>>> I have the same question. Sometimes even `./gradlew clean` fails
>>> due to failure of `setupVirtualEnv` tasks.
>>> >>>>>>
>>> >>>>>> Manu Zhang
>>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>>> wrote:
>>> >>>>>>
>>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks
>>> that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall
>>> it took 4 minutes. Is this intended?
>>> >>>>>>
>>> >>>>>> Kenn
>>>
>>
Re: gradle clean causes long-running python installs
Posted by Kenneth Knowles <kl...@google.com>.
Maybe add that case to https://issues.apache.org/jira/browse/BEAM-6459.
Kenn
On Fri, Feb 8, 2019 at 9:09 AM Thomas Weise <th...@apache.org> wrote:
> Probably related, a top level ./gradlew clean fails with the following:
>
> > Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
> The path python3.5 (from --python=python3.5) does not exist
>
> Can we just limit clean to do cleanup?!
>
>
> On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com>
> wrote:
>
>> Just some background, grpcio-tools is what's used to do the proto
>> generation. Unfortunately it's expensive to compile and doesn't
>> provide very many wheels, so we want to install it once, not every
>> time. (It's also used in more than just tests; one needs it every time
>> the .proto files change.)
>>
>> That being said, we could probably do a much cheaper clean.
>>
>> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>> >
>> > grpcio-tools could probably be moved under the "test" tag in setup.py.
>> Not sure why it has to be specified in gradle configs.
>> >
>> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>
>> >> Can you `setupVirtualEnv` just enough to run `setup.py clean` without
>> installing gcpio-tools, etc?
>> >>
>> >> Kenn
>> >>
>> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>> >>>
>> >>> setup.py has requirements like setuptools, which are installed in the
>> virtual environment.
>> >>> So even running the clean command requires the virtualenv to be set
>> up.
>> >>>
>> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>> its output directory)
>> >>>
>> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
>> wrote:
>> >>>>
>> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
>> conclusion. Doesn't require Beam knowledge so I labeled "starter".
>> >>>>
>> >>>> Kenn
>> >>>>
>> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
>> wrote:
>> >>>>>
>> >>>>> This seems to be on purpose [1]
>> >>>>>
>> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
>> probably should work around that.
>> >>>>>
>> >>>>> [1]
>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>> >>>>>
>> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
>> wrote:
>> >>>>>>
>> >>>>>> I have the same question. Sometimes even `./gradlew clean` fails
>> due to failure of `setupVirtualEnv` tasks.
>> >>>>>>
>> >>>>>> Manu Zhang
>> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>> wrote:
>> >>>>>>
>> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks
>> that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall
>> it took 4 minutes. Is this intended?
>> >>>>>>
>> >>>>>> Kenn
>>
>
Re: gradle clean causes long-running python installs
Posted by Thomas Weise <th...@apache.org>.
Probably related, a top level ./gradlew clean fails with the following:
> Task :beam-sdks-python-precommit-direct-py3:setupVirtualenv FAILED
The path python3.5 (from --python=python3.5) does not exist
Can we just limit clean to do cleanup?!
On Mon, Jan 21, 2019 at 7:58 AM Robert Bradshaw <ro...@google.com> wrote:
> Just some background, grpcio-tools is what's used to do the proto
> generation. Unfortunately it's expensive to compile and doesn't
> provide very many wheels, so we want to install it once, not every
> time. (It's also used in more than just tests; one needs it every time
> the .proto files change.)
>
> That being said, we could probably do a much cheaper clean.
>
> On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
> >
> > grpcio-tools could probably be moved under the "test" tag in setup.py.
> Not sure why it has to be specified in gradle configs.
> >
> > On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com> wrote:
> >>
> >> Can you `setupVirtualEnv` just enough to run `setup.py clean` without
> installing gcpio-tools, etc?
> >>
> >> Kenn
> >>
> >> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
> >>>
> >>> setup.py has requirements like setuptools, which are installed in the
> virtual environment.
> >>> So even running the clean command requires the virtualenv to be set up.
> >>>
> >>> A possible fix could be to skip :beam-sdks-python:cleanPython if
> setupVirtualenv has not been run. (perhaps by checking for the existence of
> its output directory)
> >>>
> >>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com>
> wrote:
> >>>>
> >>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
> conclusion. Doesn't require Beam knowledge so I labeled "starter".
> >>>>
> >>>> Kenn
> >>>>
> >>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
> wrote:
> >>>>>
> >>>>> This seems to be on purpose [1]
> >>>>>
> >>>>> AFAIU setup is done to be able to call into setup.py clean. We
> probably should work around that.
> >>>>>
> >>>>> [1]
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
> >>>>>
> >>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
> wrote:
> >>>>>>
> >>>>>> I have the same question. Sometimes even `./gradlew clean` fails
> due to failure of `setupVirtualEnv` tasks.
> >>>>>>
> >>>>>> Manu Zhang
> >>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
> wrote:
> >>>>>>
> >>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks
> that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall
> it took 4 minutes. Is this intended?
> >>>>>>
> >>>>>> Kenn
>
Re: gradle clean causes long-running python installs
Posted by Robert Bradshaw <ro...@google.com>.
Just some background, grpcio-tools is what's used to do the proto
generation. Unfortunately it's expensive to compile and doesn't
provide very many wheels, so we want to install it once, not every
time. (It's also used in more than just tests; one needs it every time
the .proto files change.)
That being said, we could probably do a much cheaper clean.
On Fri, Jan 18, 2019 at 8:56 PM Udi Meiri <eh...@google.com> wrote:
>
> grpcio-tools could probably be moved under the "test" tag in setup.py. Not sure why it has to be specified in gradle configs.
>
> On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com> wrote:
>>
>> Can you `setupVirtualEnv` just enough to run `setup.py clean` without installing gcpio-tools, etc?
>>
>> Kenn
>>
>> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>>>
>>> setup.py has requirements like setuptools, which are installed in the virtual environment.
>>> So even running the clean command requires the virtualenv to be set up.
>>>
>>> A possible fix could be to skip :beam-sdks-python:cleanPython if setupVirtualenv has not been run. (perhaps by checking for the existence of its output directory)
>>>
>>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com> wrote:
>>>>
>>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>>
>>>> Kenn
>>>>
>>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com> wrote:
>>>>>
>>>>> This seems to be on purpose [1]
>>>>>
>>>>> AFAIU setup is done to be able to call into setup.py clean. We probably should work around that.
>>>>>
>>>>> [1] https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>>
>>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com> wrote:
>>>>>>
>>>>>> I have the same question. Sometimes even `./gradlew clean` fails due to failure of `setupVirtualEnv` tasks.
>>>>>>
>>>>>> Manu Zhang
>>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>, wrote:
>>>>>>
>>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it took 4 minutes. Is this intended?
>>>>>>
>>>>>> Kenn
Re: gradle clean causes long-running python installs
Posted by Udi Meiri <eh...@google.com>.
grpcio-tools could probably be moved under the "test" tag in setup.py. Not
sure why it has to be specified in gradle configs.
On Fri, Jan 18, 2019 at 11:43 AM Kenneth Knowles <kl...@google.com> wrote:
> Can you `setupVirtualEnv` just enough to run `setup.py clean` without
> installing gcpio-tools, etc?
>
> Kenn
>
> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>
>> setup.py has requirements like setuptools, which are installed in the
>> virtual environment.
>> So even running the clean command requires the virtualenv to be set up.
>>
>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>> its output directory
>> <https://github.com/apache/beam/blob/94322f3b138d9b4d5ca69b3d18645e5eb3267b23/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1565>
>> )
>>
>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
>>> conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>
>>> Kenn
>>>
>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
>>> wrote:
>>>
>>>> This seems to be on purpose [1]
>>>>
>>>> AFAIU setup is done to be able to call into setup.py clean. We probably
>>>> should work around that.
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>
>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
>>>> wrote:
>>>>
>>>>> I have the same question. Sometimes even `./gradlew clean` fails due
>>>>> to failure of `setupVirtualEnv` tasks.
>>>>>
>>>>> Manu Zhang
>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>>>>> wrote:
>>>>>
>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
>>>>> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
>>>>> took 4 minutes. Is this intended?
>>>>>
>>>>> Kenn
>>>>>
>>>>>
Re: gradle clean causes long-running python installs
Posted by Michael Luckey <mi...@gmail.com>.
What does `setup.py clean` do anyway? Only removing a 'build' output
folder? Or something more sophisticated? Cause those pyc files still
remain... if looking into differences between a 'Gradle cleaned'
sdks/python and 'git cleaned' one...
On Fri, Jan 18, 2019 at 8:43 PM Kenneth Knowles <kl...@google.com> wrote:
> Can you `setupVirtualEnv` just enough to run `setup.py clean` without
> installing gcpio-tools, etc?
>
> Kenn
>
> On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
>
>> setup.py has requirements like setuptools, which are installed in the
>> virtual environment.
>> So even running the clean command requires the virtualenv to be set up.
>>
>> A possible fix could be to skip :beam-sdks-python:cleanPython if
>> setupVirtualenv has not been run. (perhaps by checking for the existence of
>> its output directory
>> <https://github.com/apache/beam/blob/94322f3b138d9b4d5ca69b3d18645e5eb3267b23/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1565>
>> )
>>
>> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com> wrote:
>>
>>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
>>> conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>>
>>> Kenn
>>>
>>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
>>> wrote:
>>>
>>>> This seems to be on purpose [1]
>>>>
>>>> AFAIU setup is done to be able to call into setup.py clean. We probably
>>>> should work around that.
>>>>
>>>> [1]
>>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>>
>>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
>>>> wrote:
>>>>
>>>>> I have the same question. Sometimes even `./gradlew clean` fails due
>>>>> to failure of `setupVirtualEnv` tasks.
>>>>>
>>>>> Manu Zhang
>>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>>>>> wrote:
>>>>>
>>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
>>>>> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
>>>>> took 4 minutes. Is this intended?
>>>>>
>>>>> Kenn
>>>>>
>>>>>
Re: gradle clean causes long-running python installs
Posted by Kenneth Knowles <kl...@google.com>.
Can you `setupVirtualEnv` just enough to run `setup.py clean` without
installing gcpio-tools, etc?
Kenn
On Fri, Jan 18, 2019 at 11:20 AM Udi Meiri <eh...@google.com> wrote:
> setup.py has requirements like setuptools, which are installed in the
> virtual environment.
> So even running the clean command requires the virtualenv to be set up.
>
> A possible fix could be to skip :beam-sdks-python:cleanPython if
> setupVirtualenv has not been run. (perhaps by checking for the existence of
> its output directory
> <https://github.com/apache/beam/blob/94322f3b138d9b4d5ca69b3d18645e5eb3267b23/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1565>
> )
>
> On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com> wrote:
>
>> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
>> conclusion. Doesn't require Beam knowledge so I labeled "starter".
>>
>> Kenn
>>
>> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
>> wrote:
>>
>>> This seems to be on purpose [1]
>>>
>>> AFAIU setup is done to be able to call into setup.py clean. We probably
>>> should work around that.
>>>
>>> [1]
>>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>>
>>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
>>> wrote:
>>>
>>>> I have the same question. Sometimes even `./gradlew clean` fails due to
>>>> failure of `setupVirtualEnv` tasks.
>>>>
>>>> Manu Zhang
>>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>>>> wrote:
>>>>
>>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
>>>> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
>>>> took 4 minutes. Is this intended?
>>>>
>>>> Kenn
>>>>
>>>>
Re: gradle clean causes long-running python installs
Posted by Udi Meiri <eh...@google.com>.
setup.py has requirements like setuptools, which are installed in the
virtual environment.
So even running the clean command requires the virtualenv to be set up.
A possible fix could be to skip :beam-sdks-python:cleanPython if
setupVirtualenv has not been run. (perhaps by checking for the existence of
its output directory
<https://github.com/apache/beam/blob/94322f3b138d9b4d5ca69b3d18645e5eb3267b23/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1565>
)
On Wed, Jan 16, 2019 at 7:03 PM Kenneth Knowles <kl...@google.com> wrote:
> Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
> conclusion. Doesn't require Beam knowledge so I labeled "starter".
>
> Kenn
>
> On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com>
> wrote:
>
>> This seems to be on purpose [1]
>>
>> AFAIU setup is done to be able to call into setup.py clean. We probably
>> should work around that.
>>
>> [1]
>> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>>
>> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
>> wrote:
>>
>>> I have the same question. Sometimes even `./gradlew clean` fails due to
>>> failure of `setupVirtualEnv` tasks.
>>>
>>> Manu Zhang
>>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>>> wrote:
>>>
>>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
>>> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
>>> took 4 minutes. Is this intended?
>>>
>>> Kenn
>>>
>>>
Re: gradle clean causes long-running python installs
Posted by Kenneth Knowles <kl...@google.com>.
Filed https://issues.apache.org/jira/browse/BEAM-6459 to record the
conclusion. Doesn't require Beam knowledge so I labeled "starter".
Kenn
On Wed, Jan 16, 2019 at 12:14 AM Michael Luckey <ad...@gmail.com> wrote:
> This seems to be on purpose [1]
>
> AFAIU setup is done to be able to call into setup.py clean. We probably
> should work around that.
>
> [1]
> https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
>
> On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com>
> wrote:
>
>> I have the same question. Sometimes even `./gradlew clean` fails due to
>> failure of `setupVirtualEnv` tasks.
>>
>> Manu Zhang
>> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>,
>> wrote:
>>
>> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
>> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
>> took 4 minutes. Is this intended?
>>
>> Kenn
>>
>>
Re: gradle clean causes long-running python installs
Posted by Michael Luckey <ad...@gmail.com>.
This seems to be on purpose [1]
AFAIU setup is done to be able to call into setup.py clean. We probably
should work around that.
[1]
https://github.com/apache/beam/blob/master/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L1600-L1610
On Wed, Jan 16, 2019 at 7:01 AM Manu Zhang <ow...@gmail.com> wrote:
> I have the same question. Sometimes even `./gradlew clean` fails due to
> failure of `setupVirtualEnv` tasks.
>
> Manu Zhang
> On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>, wrote:
>
> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that
> invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it
> took 4 minutes. Is this intended?
>
> Kenn
>
>
Re: gradle clean causes long-running python installs
Posted by Manu Zhang <ow...@gmail.com>.
I have the same question. Sometimes even `./gradlew clean` fails due to failure of `setupVirtualEnv` tasks.
Manu Zhang
On Jan 16, 2019, 12:22 PM +0800, Kenneth Knowles <ke...@apache.org>, wrote:
> A global `./gradlew clean` runs various `setupVirtualEnv` tasks that invoke things such as `setup.py bdist_wheel for grpcio-tools`. Overall it took 4 minutes. Is this intended?
>
> Kenn