You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nguymin4 (via GitHub)" <gi...@apache.org> on 2025/06/21 19:40:59 UTC
[PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
nguymin4 opened a new pull request, #35388:
URL: https://github.com/apache/beam/pull/35388
Support Docker Compose v2 in Python TransformServiceLauncher
From July 2023, Compose v1 (`docker-compose`) stopped receiving updates. Probably it's time to migrate to modern Compose v2 (`docker compose`).
This PR introduce a flag `use_docker_compose_v2` with default value `False` to maintain compatibility.
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #35388:
URL: https://github.com/apache/beam/pull/35388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2193407144
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
@tvalentyn I have no strong opinion about this. I currently put it here as it's quite clear about the checking steps: java -> docker -> docker compose.
But I think it's ok to put it inside `transform_service_launcher.py` as well
Should I make the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-2994021274
assign set of reviewers
Note: Current CI failed due to different reason
```
Execution failed for task ':sdks:java:extensions:sql:generateFmppSources'.
> Could not resolve all files for configuration ':sdks:java:extensions:sql:fmppTask'.
> Could not resolve oro:oro:[2.0.8,).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-2994022041
Assigning reviewers:
R: @tvalentyn for label python.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-3016642890
Reminder, please take a look at this pr: @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-2993781596
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-3027652835
Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @shunping for label python.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on PR #35388:
URL: https://github.com/apache/beam/pull/35388#issuecomment-2994023143
Current CI failed due to different reason
```
Execution failed for task ':sdks:java:extensions:sql:generateFmppSources'.
> Could not resolve all files for configuration ':sdks:java:extensions:sql:fmppTask'.
> Could not resolve oro:oro:[2.0.8,).
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2195432395
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
I am not very familiar with the control flow here and didn't look very closely, but if we can infer the settings closer to where docker is invoked, that seems preferable to plumbing along the setting through a few calls in the stack.
We do need to make sure that the code where we check for docker settings is running in the envrionment that has docker, where the command will actually be running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2204196689
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
@tvalentyn How does the new change look to you?
CI failed due to unrelated reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2204196689
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
@tvalentyn How does the new change look to you?
CI failed due to unrelated reason `Unable to start redis container for RRIO tests ..... no space left on device: exit status 1: unknown`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2204196689
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
@tvalentyn How does the new change look to you?
CI failed due to unrelated reason `no space left on device: exit status 1: unknown`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2193029707
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
Does it make sense to make this check in transform_service_launcher.py instead?
Also cc: @chamikaramj
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2193407144
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
@tvalentyn I have no strong opinion about this. I currently put it here as it's quite clear about the checking steps: java -> docker -> docker compose.
Should I make the change?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2195432395
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
I am not very familiar with the control flow here and didn't look very closely, but if we can infer the setting closer to where docker is invoked, that seems preferable to plumbing along the setting through a few calls in the stack.
We do need to make sure that the code where we check for docker settings is running in the envrionment that has docker, where the command will actually be running.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Support Docker Compose v2 in Python TransformServiceLauncher [beam]
Posted by "nguymin4 (via GitHub)" <gi...@apache.org>.
nguymin4 commented on code in PR #35388:
URL: https://github.com/apache/beam/pull/35388#discussion_r2195932315
##########
sdks/python/apache_beam/transforms/external.py:
##########
@@ -1174,6 +1174,16 @@ def is_docker_available():
return True
+ def is_docker_compose_v2_available():
+ cmd = ['docker', 'compose', 'version']
+
+ try:
+ subprocess.Popen(cmd, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
Review Comment:
I made my change in this commit [e337a1a](https://github.com/apache/beam/pull/35388/commits/e337a1a7a2193ccbe871c2ee10e227926a7e9dbc)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org