You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Heejong Lee <he...@google.com> on 2022/01/26 23:15:43 UTC
[FYI] -PpythonVersion might have not worked as expected
Hi,
If you've ever encountered the Python test flakiness related to Python
versions (-PpythonVersion), this might be the root cause:
*Summary*
An existing virtualenv directory should be cleared before creating a new
one.
*Problem Description*
A virtualenv directory name for Python tasks is generated from the hash of
the project path so any tasks that have the same project path share the
same virtualenv directory. The problem is that when the setupVirtualenv
task initializes a new virtualenv directory it doesn't overwrite existing
data. This can cause a subtle bug which is very hard to debug. See the
following example:
❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.8*
Configuration on demand is an incubating feature.
> Task :sdks:python:setupVirtualenv
> Task :sdks:python:sdist
> Task :sdks:python:installGcpTest
> Task :sdks:python:wordCount
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:oauth2client.transport:Attempting refresh to obtain initial
access_token
INFO:oauth2client.client:Refreshing access_token
WARNING:root:Make sure that locally built Python SDK docker image has
Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is
*apache/beam_python3.8_sdk:2.37.0.dev
<http://2.37.0.dev>*
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x122f479d0>
====================
BUILD SUCCESSFUL in 1m 14s
14 actionable tasks: 4 executed, 10 up-to-date
❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.6*
Configuration on demand is an incubating feature.
> Task :sdks:python:setupVirtualenv
> Task :sdks:python:installGcpTest
> Task :sdks:python:wordCount
INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
seconds.
INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
INFO:oauth2client.transport:Attempting refresh to obtain initial
access_token
INFO:oauth2client.client:Refreshing access_token
WARNING:root:Make sure that locally built Python SDK docker image has
Python 3.8 interpreter.
INFO:root:Default Python SDK image for environment is
*apache/beam_python3.8_sdk:2.37.0.dev
<http://2.37.0.dev>*
INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
<function annotate_downstream_side_inputs at 0x124afa9d0>
====================
BUILD SUCCESSFUL in 1m 8s
14 actionable tasks: 3 executed, 11 up-to-date
Note that the second Gradle command specified Python 3.6 but the executed
test adopted Python 3.8. The first Python version used right after the
clean task fixes the virtualenv Python version. Any tasks thereafter based
on the same project path will use the first Python version as shown above.
*Affected Tests*
We have Python test suites that run against multiple Python versions.
Luckily, most of them have Python versions as a part of their project paths
e.g. :sdks:python:test-suites:dataflow:py38:setupVirtualenv. For automated
Jenkins tests, we also utilize tasks created for each Python version. The
only exception is cross-language tests which use for-each loop to run the
multiple tests for each target Python version. In summary:
- Jenkins Python tests were not affected. In other words, we have good
coverage for multiple Python versions
- Cross-language VR tests were affected. It means that we missed the
test coverage of the second Python version, namely Python 3.8
- Any tests executed directly from the command-line could be error-prone
since -PpythonVersion flag only works for the first task on the pristine
source directory
*Solution*
venv module supports --clear option which removes any existing virtualenv
directory before initializing a new one.
*Links*
Jira: https://issues.apache.org/jira/browse/BEAM-13716
PR: https://github.com/apache/beam/pull/16591
Re: [FYI] -PpythonVersion might have not worked as expected
Posted by Valentyn Tymofieiev <va...@google.com>.
Thanks, Heejong, great sleuthing!
Few thoughts:
- Gradle runs targets in the same directory sequentially, so --clear option
shouldn't have negative consequences on concurrent run.
- This is also why we have py36, py37, py38 directories for some suites -
because we wanted to be able to run them in parallel.
- As a reminder for everyone, please don't hardcode Python versions for
your test suites inside the suites. Instead, see the top-level
configuration files [1], [2] depending on how your suite is configured.
[1]
https://github.com/apache/beam/blob/master/sdks/python/test-suites/gradle.properties
[2]
https://github.com/apache/beam/blob/f72f1972ce18c467f9080d4952315e998cdfe0f8/.test-infra/jenkins/PythonTestProperties.groovy#L36
On Wed, Jan 26, 2022 at 4:06 PM Ahmet Altay <al...@google.com> wrote:
> Thank you for the detailed information Heejong.
>
> On Wed, Jan 26, 2022 at 3:16 PM Heejong Lee <he...@google.com> wrote:
>
>> Hi,
>>
>> If you've ever encountered the Python test flakiness related to Python
>> versions (-PpythonVersion), this might be the root cause:
>>
>> *Summary*
>>
>> An existing virtualenv directory should be cleared before creating a new
>> one.
>>
>> *Problem Description*
>>
>> A virtualenv directory name for Python tasks is generated from the hash
>> of the project path so any tasks that have the same project path share the
>> same virtualenv directory. The problem is that when the setupVirtualenv
>> task initializes a new virtualenv directory it doesn't overwrite existing
>> data. This can cause a subtle bug which is very hard to debug. See the
>> following example:
>>
>> ❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.8*
>> Configuration on demand is an incubating feature.
>>
>> > Task :sdks:python:setupVirtualenv
>>
>> > Task :sdks:python:sdist
>>
>> > Task :sdks:python:installGcpTest
>>
>> > Task :sdks:python:wordCount
>> INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
>> seconds.
>> INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
>> INFO:oauth2client.transport:Attempting refresh to obtain initial
>> access_token
>> INFO:oauth2client.client:Refreshing access_token
>> WARNING:root:Make sure that locally built Python SDK docker image has
>> Python 3.8 interpreter.
>> INFO:root:Default Python SDK image for environment is *apache/beam_python3.8_sdk:2.37.0.dev
>> <http://2.37.0.dev>*
>> INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
>> <function annotate_downstream_side_inputs at 0x122f479d0>
>> ====================
>>
>> BUILD SUCCESSFUL in 1m 14s
>> 14 actionable tasks: 4 executed, 10 up-to-date
>>
>> ❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.6*
>> Configuration on demand is an incubating feature.
>>
>> > Task :sdks:python:setupVirtualenv
>>
>> > Task :sdks:python:installGcpTest
>>
>> > Task :sdks:python:wordCount
>> INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
>> seconds.
>> INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
>> INFO:oauth2client.transport:Attempting refresh to obtain initial
>> access_token
>> INFO:oauth2client.client:Refreshing access_token
>> WARNING:root:Make sure that locally built Python SDK docker image has
>> Python 3.8 interpreter.
>> INFO:root:Default Python SDK image for environment is *apache/beam_python3.8_sdk:2.37.0.dev
>> <http://2.37.0.dev>*
>> INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
>> <function annotate_downstream_side_inputs at 0x124afa9d0>
>> ====================
>>
>> BUILD SUCCESSFUL in 1m 8s
>> 14 actionable tasks: 3 executed, 11 up-to-date
>>
>> Note that the second Gradle command specified Python 3.6 but the executed
>> test adopted Python 3.8. The first Python version used right after the
>> clean task fixes the virtualenv Python version. Any tasks thereafter based
>> on the same project path will use the first Python version as shown above.
>>
>> *Affected Tests*
>> We have Python test suites that run against multiple Python versions.
>> Luckily, most of them have Python versions as a part of their project paths
>> e.g. :sdks:python:test-suites:dataflow:py38:setupVirtualenv. For
>> automated Jenkins tests, we also utilize tasks created for each Python
>> version. The only exception is cross-language tests which use for-each loop
>> to run the multiple tests for each target Python version. In summary:
>>
>>
>> - Jenkins Python tests were not affected. In other words, we have
>> good coverage for multiple Python versions
>> - Cross-language VR tests were affected. It means that we missed the
>> test coverage of the second Python version, namely Python 3.8
>> - Any tests executed directly from the command-line could be
>> error-prone since -PpythonVersion flag only works for the first task
>> on the pristine source directory
>>
>> *Solution*
>> venv module supports --clear option which removes any existing
>> virtualenv directory before initializing a new one.
>>
>> *Links*
>> Jira: https://issues.apache.org/jira/browse/BEAM-13716
>> PR: https://github.com/apache/beam/pull/16591
>>
>
Re: [FYI] -PpythonVersion might have not worked as expected
Posted by Ahmet Altay <al...@google.com>.
Thank you for the detailed information Heejong.
On Wed, Jan 26, 2022 at 3:16 PM Heejong Lee <he...@google.com> wrote:
> Hi,
>
> If you've ever encountered the Python test flakiness related to Python
> versions (-PpythonVersion), this might be the root cause:
>
> *Summary*
>
> An existing virtualenv directory should be cleared before creating a new
> one.
>
> *Problem Description*
>
> A virtualenv directory name for Python tasks is generated from the hash of
> the project path so any tasks that have the same project path share the
> same virtualenv directory. The problem is that when the setupVirtualenv
> task initializes a new virtualenv directory it doesn't overwrite existing
> data. This can cause a subtle bug which is very hard to debug. See the
> following example:
>
> ❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.8*
> Configuration on demand is an incubating feature.
>
> > Task :sdks:python:setupVirtualenv
>
> > Task :sdks:python:sdist
>
> > Task :sdks:python:installGcpTest
>
> > Task :sdks:python:wordCount
> INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
> seconds.
> INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
> INFO:oauth2client.transport:Attempting refresh to obtain initial
> access_token
> INFO:oauth2client.client:Refreshing access_token
> WARNING:root:Make sure that locally built Python SDK docker image has
> Python 3.8 interpreter.
> INFO:root:Default Python SDK image for environment is *apache/beam_python3.8_sdk:2.37.0.dev
> <http://2.37.0.dev>*
> INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
> <function annotate_downstream_side_inputs at 0x122f479d0>
> ====================
>
> BUILD SUCCESSFUL in 1m 14s
> 14 actionable tasks: 4 executed, 10 up-to-date
>
> ❯ ./gradlew :sdks:python:wordCount *-PpythonVersion=3.6*
> Configuration on demand is an incubating feature.
>
> > Task :sdks:python:setupVirtualenv
>
> > Task :sdks:python:installGcpTest
>
> > Task :sdks:python:wordCount
> INFO:apache_beam.internal.gcp.auth:Setting socket default timeout to 60
> seconds.
> INFO:apache_beam.internal.gcp.auth:socket default timeout is 60.0 seconds.
> INFO:oauth2client.transport:Attempting refresh to obtain initial
> access_token
> INFO:oauth2client.client:Refreshing access_token
> WARNING:root:Make sure that locally built Python SDK docker image has
> Python 3.8 interpreter.
> INFO:root:Default Python SDK image for environment is *apache/beam_python3.8_sdk:2.37.0.dev
> <http://2.37.0.dev>*
> INFO:apache_beam.runners.portability.fn_api_runner.translations:====================
> <function annotate_downstream_side_inputs at 0x124afa9d0>
> ====================
>
> BUILD SUCCESSFUL in 1m 8s
> 14 actionable tasks: 3 executed, 11 up-to-date
>
> Note that the second Gradle command specified Python 3.6 but the executed
> test adopted Python 3.8. The first Python version used right after the
> clean task fixes the virtualenv Python version. Any tasks thereafter based
> on the same project path will use the first Python version as shown above.
>
> *Affected Tests*
> We have Python test suites that run against multiple Python versions.
> Luckily, most of them have Python versions as a part of their project paths
> e.g. :sdks:python:test-suites:dataflow:py38:setupVirtualenv. For
> automated Jenkins tests, we also utilize tasks created for each Python
> version. The only exception is cross-language tests which use for-each loop
> to run the multiple tests for each target Python version. In summary:
>
>
> - Jenkins Python tests were not affected. In other words, we have good
> coverage for multiple Python versions
> - Cross-language VR tests were affected. It means that we missed the
> test coverage of the second Python version, namely Python 3.8
> - Any tests executed directly from the command-line could be
> error-prone since -PpythonVersion flag only works for the first task
> on the pristine source directory
>
> *Solution*
> venv module supports --clear option which removes any existing virtualenv
> directory before initializing a new one.
>
> *Links*
> Jira: https://issues.apache.org/jira/browse/BEAM-13716
> PR: https://github.com/apache/beam/pull/16591
>