You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Abacn (via GitHub)" <gi...@apache.org> on 2023/05/26 21:22:57 UTC
[GitHub] [beam] Abacn commented on a diff in pull request #26913: Always install dev wheel if passed to sdk_version
Abacn commented on code in PR #26913:
URL: https://github.com/apache/beam/pull/26913#discussion_r1207375171
##########
sdks/python/container/piputil.go:
##########
@@ -148,8 +148,11 @@ func findBeamSdkWhl(files []string, acceptableWhlSpecs []string) string {
// SDK from source tarball provided in sdkSrcFile.
func installSdk(files []string, workDir string, sdkSrcFile string, acceptableWhlSpecs []string, required bool) error {
sdkWhlFile := findBeamSdkWhl(files, acceptableWhlSpecs)
+
if sdkWhlFile != "" {
- err := pipInstallPackage(files, workDir, sdkWhlFile, false, false, []string{"gcp"})
+ // by default, pip rejects to install wheel if same version already installed
+ isDev := strings.Contains(sdkWhlFile, ".dev")
Review Comment:
tarball is not affected. checked the log if passing a tarball (same pipeline options above expect sdk_location)
```
DEBUG 2023-05-26T21:15:34.103230283Z Found artifact: dataflow_python_sdk.tar
INFO 2023-05-26T21:15:34.103325167Z 2023/05/26 21:15:34 Installing setup packages ...
INFO 2023-05-26T21:15:34.645905671Z Processing /var/opt/google/staged/dataflow_python_sdk.tar
INFO 2023-05-26T21:15:35.031355177Z Preparing metadata (setup.py): started
...
INFO 2023-05-26T21:16:30.151314981Z Successfully built apache-beam
INFO 2023-05-26T21:16:31.978811453Z Installing collected packages: apache-beam
INFO 2023-05-26T21:16:31.979188478Z Attempting uninstall: apache-beam
INFO 2023-05-26T21:16:31.989685770Z Found existing installation: apache-beam 2.49.0.dev0
...
INFO 2023-05-26T21:16:37.762490639Z Successfully installed apache-beam-2.49.0.dev0
DEBUG 2023-05-26T21:16:38.099652894Z Executing Python (worker sdk-0-0_sibling_1): python -m apache_beam.runners.worker.sdk_worker_main
DEBUG 2023-05-26T21:16:38.268671515Z Executing Python (worker sdk-0-0): python -m apache_beam.runners.worker.sdk_worker_main
```
btw now install from tarball costs 1 minute with sibling worker enabled (previously took 3.5 min when every cpu core run a container)
though it seems PostCommit test still not run with sibling worker
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org