You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "ahmedabu98 (via GitHub)" <gi...@apache.org> on 2023/05/26 13:56:19 UTC
[GitHub] [beam] ahmedabu98 opened a new issue, #26907: [Bug]: Getting Python SDK location at runtime via wheels does not update to the new release
ahmedabu98 opened a new issue, #26907:
URL: https://github.com/apache/beam/issues/26907
### What happened?
[beam_PostCommit_Python_Xlang_Gcp_Dataflow](https://ci-beam.apache.org/job/beam_PostCommit_Python_Xlang_Gcp_Dataflow/) tests started failing when release 2.48.0 was cut with the following error:
```
RuntimeError: Pipeline construction environment and pipeline runtime environment are not compatible.
If you use a custom container image, check that the Python interpreter minor version and the Apache
Beam version in your image match the versions used at pipeline construction time.
Submission environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.49.0.dev.
Runtime environment: beam:version:sdk_base:apache/beam_python3.8_sdk:2.48.0.dev.
```
This is weird because the sdk location provided at runtime is labeled `2.49.0.dev0`:
`--sdk_location=/home/jenkins/jenkins-slave/workspace/beam_PostCommit_Python_Xlang_Gcp_Dataflow_PR/src/sdks/python/build/apache_beam-2.49.0.dev0-cp37-cp37m-manylinux_2_17_x86_64.manylinux2014_x86_64.whl`
These tests get their sdk_location at runtime via wheels:
https://github.com/apache/beam/blob/31c3390bbb4c7e6f1fc7f2c366b20d49556a61c8/sdks/python/test-suites/dataflow/common.gradle#L40-L52
After deleting that portion of code to force using tarball, the tests pass.
I've noticed that other xlang tests haven't run into this and suspect if this is because they don't use wheels to get sdk_location:
Load tests only use tarball:
https://github.com/apache/beam/blob/675d509967b7dc988aefb54181319fec237d4d06/sdks/python/apache_beam/testing/load_tests/build.gradle#L42-L45
XVR tests use the following:
https://github.com/apache/beam/blob/0963cf0bd0a791f86c1353ad488fe28f99cdfd0c/buildSrc/src/main/groovy/org/apache/beam/gradle/BeamModulePlugin.groovy#L2597-L2603
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn closed issue #26907: [Bug]: Wheels Python SDK location does not update to the new release
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #26907: [Bug]: Wheels Python SDK location does not update to the new release
URL: https://github.com/apache/beam/issues/26907
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26907: [Bug]: Wheels Python SDK location does not update to the new release
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26907:
URL: https://github.com/apache/beam/issues/26907#issuecomment-1564457136
Before release cut:
<img width="1346" alt="image" src="https://github.com/apache/beam/assets/8010435/3505ff4e-6c72-4f55-a76e-0b24c42551b8">
After release cut:
<img width="1345" alt="image" src="https://github.com/apache/beam/assets/8010435/adb9e975-d8da-46d9-bd65-442edb2ab2f4">
There is one log missing that is from here:
https://github.com/apache/beam/blob/170b597b5eaa56ffd5ac3c97514e4e8be277cb46/sdks/python/container/boot.go#L368
Somehow the SDK is not installed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] Abacn commented on issue #26907: [Bug]: Wheels Python SDK location does not update to the new release
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #26907:
URL: https://github.com/apache/beam/issues/26907#issuecomment-1564448377
This is because the test did not specify `--sdk_container_image` pipeline option for Python container image, and falls back to use beam-master container defined in https://github.com/apache/beam/blob/170b597b5eaa56ffd5ac3c97514e4e8be277cb46/sdks/python/apache_beam/runners/dataflow/internal/names.py#L42
which has 2.48.0-dev0 version.
Then the container somehow forbids installing a different version of beam via wheel, but does not check tarball on container startup.
Include `--sdk_container_image` in test should resolve the issue
https://github.com/apache/beam/blob/170b597b5eaa56ffd5ac3c97514e4e8be277cb46/sdks/python/apache_beam/runners/dataflow/internal/names.py#L42
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org