You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "riteshghorse (via GitHub)" <gi...@apache.org> on 2023/08/17 15:58:26 UTC

[GitHub] [beam] riteshghorse opened a new pull request, #28046: [Python] Allow configuring Python SDK harness boot entry point with Env variable

riteshghorse opened a new pull request, #28046:
URL: https://github.com/apache/beam/pull/28046

   Use `GetPythonVersion` utility to find the available python command to use. If `PYTHON_PATH` is set then it uses that.
   
   Fixes #26958 
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1690314515

   Run PythonDocs PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1686921298

   Run PythonDocs PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28046:
URL: https://github.com/apache/beam/pull/28046#discussion_r1300663242


##########
sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go:
##########
@@ -396,7 +396,9 @@ func GetPythonVersion() (string, error) {
 			return v, nil
 		}
 	}
-	return "", fmt.Errorf("no python installation found")
+	return "", errors.New("no python installation found. If you use a" +

Review Comment:
   add spaces before closing quotes.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on a diff in pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #28046:
URL: https://github.com/apache/beam/pull/28046#discussion_r1300506160


##########
sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download.go:
##########
@@ -382,7 +382,14 @@ func jarExists(jarPath string) bool {
 	return err == nil
 }
 
-func getPythonVersion() (string, error) {
+// GetPythonVersion returns the Python version to use. It checks for
+// env variable PYTHON_PATH and returns that it if set.
+// If no PYTHON_PATH is defined then it checks for python or python3

Review Comment:
   Let's modify the error message on line 399 to say sth like: 
   
   no python installation found. If you use a custom container image, please verify that python or python3 command launches the interpreter  or specify the full path to the interpreter file in PYTHON_PATH environment variable.



##########
sdks/python/container/boot.go:
##########
@@ -91,7 +92,12 @@ func main() {
 			"--container_executable=/opt/apache/beam/boot",
 		}
 		log.Printf("Starting worker pool %v: python %v", workerPoolId, strings.Join(args, " "))
-		if err := execx.Execute("python", args...); err != nil {
+		pythonVersion, err := expansionx.GetPythonVersion()
+		if err != nil {
+			log.Print("Python SDK worker pool exited.")

Review Comment:
   Can we buble up the "no python installation found" error instead?
   Should we use  `log.Fatalf` instead?



##########
sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download_test.go:
##########
@@ -217,3 +217,31 @@ func TestGetJar_dev(t *testing.T) {
 		t.Errorf("error message does not contain gradle command %v for user, got message: %v", gradleTarget, err)
 	}
 }
+
+func TestGetPythonVersion(t *testing.T) {
+	tests := []struct {
+		name        string
+		PYTHON_PATH string
+	}{
+		{
+			name:        "PYTHON_PATH set",
+			PYTHON_PATH: "/bin/python",
+		},
+		{
+			name:        "PYTHON_PATH not set",
+			PYTHON_PATH: "",
+		},
+	}
+
+	for _, test := range tests {
+		os.Setenv("PYTHON_PATH", test.PYTHON_PATH)

Review Comment:
   Consider setting only if not empty.



##########
sdks/python/container/boot.go:
##########
@@ -336,7 +342,11 @@ func setupVenv(ctx context.Context, logger *tools.Logger, baseDir, workerId stri
 	if err := os.MkdirAll(dir, 0750); err != nil {
 		return "", fmt.Errorf("failed to create Python venv directory: %s", err)
 	}
-	if err := execx.Execute("python", "-m", "venv", "--system-site-packages", dir); err != nil {
+	pythonVersion, err := expansionx.GetPythonVersion()
+	if err != nil {
+		return "", fmt.Errorf("python installation not found: %v", err)

Review Comment:
   should we return `err` without modifications here this message is already included?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse merged pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse merged PR #28046:
URL: https://github.com/apache/beam/pull/28046


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1686840066

   > Oh I think we would need to add a flag to configure python path. 
   
   You mean a flag for boot.go? I think having env variable is good enough.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Allow configuring Python SDK harness boot entry point with Env variable

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1682536991

   R: @tvalentyn 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1688239226

   Run Python Dataflow ValidatesContainer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on a diff in pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28046:
URL: https://github.com/apache/beam/pull/28046#discussion_r1300545227


##########
sdks/python/container/boot.go:
##########
@@ -336,7 +342,11 @@ func setupVenv(ctx context.Context, logger *tools.Logger, baseDir, workerId stri
 	if err := os.MkdirAll(dir, 0750); err != nil {
 		return "", fmt.Errorf("failed to create Python venv directory: %s", err)
 	}
-	if err := execx.Execute("python", "-m", "venv", "--system-site-packages", dir); err != nil {
+	pythonVersion, err := expansionx.GetPythonVersion()
+	if err != nil {
+		return "", fmt.Errorf("python installation not found: %v", err)

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1686874335

   no, it would have to be set in Dockerfile using `ENV` command.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1686915062

   > > > Oh I think we would need to add a flag to configure python path.
   > > 
   > > 
   > > You mean a flag for boot.go? I think having env variable is good enough.
   > 
   > Got it. I was wondering if the flag set on local machine would be picked up on the worker.
   
   OK, I'll send a CL to update the custom container doc as well near 2.51.0 release


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1688616638

   Run PythonDocs PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] github-actions[bot] commented on pull request #28046: [Python] Allow configuring Python SDK harness boot entry point with Env variable

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1682540178

   Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1688352810

   Run Python_Runners PreCommit


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1689070266

   Run Python Dataflow ValidatesContainer


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on a diff in pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28046:
URL: https://github.com/apache/beam/pull/28046#discussion_r1300549706


##########
sdks/go/pkg/beam/core/runtime/xlangx/expansionx/download_test.go:
##########
@@ -217,3 +217,31 @@ func TestGetJar_dev(t *testing.T) {
 		t.Errorf("error message does not contain gradle command %v for user, got message: %v", gradleTarget, err)
 	}
 }
+
+func TestGetPythonVersion(t *testing.T) {
+	tests := []struct {
+		name        string
+		PYTHON_PATH string
+	}{
+		{
+			name:        "PYTHON_PATH set",
+			PYTHON_PATH: "/bin/python",
+		},
+		{
+			name:        "PYTHON_PATH not set",
+			PYTHON_PATH: "",
+		},
+	}
+
+	for _, test := range tests {
+		os.Setenv("PYTHON_PATH", test.PYTHON_PATH)

Review Comment:
   done



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1682670419

   Oh I think we would need to add a flag to configure python path. But this PR just adds the utility to read that env variable and choosing available python version otherwise


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] codecov[bot] commented on pull request #28046: [Python] Allow configuring Python SDK harness boot entry point with Env variable

Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1682562404

   ## [Codecov](https://app.codecov.io/gh/apache/beam/pull/28046?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
   > Merging [#28046](https://app.codecov.io/gh/apache/beam/pull/28046?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (d18341c) into [master](https://app.codecov.io/gh/apache/beam/commit/5fbc699cfb5e0441796b85a5dc93dd1371b391b5?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) (5fbc699) will **decrease** coverage by `0.01%`.
   > Report is 12 commits behind head on master.
   > The diff coverage is `80.00%`.
   
   ```diff
   @@            Coverage Diff             @@
   ##           master   #28046      +/-   ##
   ==========================================
   - Coverage   72.32%   72.31%   -0.01%     
   ==========================================
     Files         678      678              
     Lines       99740    99743       +3     
   ==========================================
   - Hits        72138    72133       -5     
   - Misses      26039    26050      +11     
   + Partials     1563     1560       -3     
   ```
   
   | [Flag](https://app.codecov.io/gh/apache/beam/pull/28046/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [go](https://app.codecov.io/gh/apache/beam/pull/28046/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `53.69% <80.00%> (-0.02%)` | :arrow_down: |
   
   Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
   
   | [Files Changed](https://app.codecov.io/gh/apache/beam/pull/28046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
   |---|---|---|
   | [...kg/beam/core/runtime/xlangx/expansionx/download.go](https://app.codecov.io/gh/apache/beam/pull/28046?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUveGxhbmd4L2V4cGFuc2lvbngvZG93bmxvYWQuZ28=) | `16.55% <80.00%> (+2.90%)` | :arrow_up: |
   
   ... and [2 files with indirect coverage changes](https://app.codecov.io/gh/apache/beam/pull/28046/indirect-changes?src=pr&el=tree-more&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   
   :mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1687071896

   please run validates container suite. It would be nice to also and test the negative case by building a container image that doesn't have a python interpreter and use an updated boot.go code + run on Dataflow. Pipeline will stall for ~1hr (expected), but there should be error messages in worker-starter or worker logs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1689069588

   > If the python interpreter is not available then it will fail at the pip install apache-beam[gcp] step itself when building the container
   
   you can build a regular container, then manually build another container:
   
   ```
   FROM buster
   COPY --from=<your_container_image> /opt/apache/beam /opt/apache/beam
   
   ENTRYPOINT ["/opt/apache/beam/boot"]
   ```


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on a diff in pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on code in PR #28046:
URL: https://github.com/apache/beam/pull/28046#discussion_r1300544741


##########
sdks/python/container/boot.go:
##########
@@ -91,7 +92,12 @@ func main() {
 			"--container_executable=/opt/apache/beam/boot",
 		}
 		log.Printf("Starting worker pool %v: python %v", workerPoolId, strings.Join(args, " "))
-		if err := execx.Execute("python", args...); err != nil {
+		pythonVersion, err := expansionx.GetPythonVersion()
+		if err != nil {
+			log.Print("Python SDK worker pool exited.")

Review Comment:
   yeah



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1686868198

   > > Oh I think we would need to add a flag to configure python path.
   > 
   > You mean a flag for boot.go? I think having env variable is good enough.
   
   Got it. I was wondering if the flag set on local machine would be picked up on the worker. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1687030987

   Failing tests are not related to this PR


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1688615499

   Without python inter
   
   > 
   
   If the python interpreter is not available then it will fail at the `pip install apache-beam[gcp]` step itself when building the container


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] tvalentyn commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1689070788

   VC run passed before: https://ci-beam.apache.org/job/beam_PostCommit_Py_ValCont_PR/306/


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1690187012

   I'll give a try with custom container image and merge this after that


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] riteshghorse commented on pull request #28046: [Python] Get available python version and use it for Python SDK harness boot entry point

Posted by "riteshghorse (via GitHub)" <gi...@apache.org>.
riteshghorse commented on PR #28046:
URL: https://github.com/apache/beam/pull/28046#issuecomment-1690314091

   The error message we wrote is logged for this [job](https://pantheon.corp.google.com/dataflow/jobs/us-central1/2023-08-23_09_49_03-4656118819153030201;bottomTab=WORKER_LOGS;graphView=0;logsSeverity=INFO?pageState=(%22dfTime%22:(%22s%22:%222023-08-23T16:49:04.304Z%22,%22e%22:%222023-08-23T16:49:23.391Z%22))&project=google.com:clouddfe)
   
   <img width="1449" alt="image" src="https://github.com/apache/beam/assets/25881114/b8e010c6-e6e3-4f11-b563-92621c3b4eb3">
   
   using the dockerfile:
   ```
   FROM debian
   
   COPY --from=us.gcr.io/google.com/clouddfe/riteshghorse/beam_python3.9_sdk:python_custom /opt/apache/beam /opt/apache/beam
   
   ENTRYPOINT ["/opt/apache/beam/boot"]
   ```
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org