You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "mohamedawnallah (via GitHub)" <gi...@apache.org> on 2025/07/14 16:38:13 UTC
[PR] workflows+gradle: run ML tests that require non self-hosted environment [beam]
mohamedawnallah opened a new pull request, #35585:
URL: https://github.com/apache/beam/pull/35585
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.
mohamedawnallah commented on PR #35585:
URL: https://github.com/apache/beam/pull/35585#issuecomment-3070440666
cc @damccorm for thoughts and feedback on this approach 🙏
PS:
`beam_PreCommit_Python_Runners` CI failure not related to the changes this PR introduce
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "mohamedawnallah (via GitHub)" <gi...@apache.org>.
mohamedawnallah commented on code in PR #35585:
URL: https://github.com/apache/beam/pull/35585#discussion_r2205723348
##########
sdks/python/pytest.ini:
##########
@@ -70,6 +70,7 @@ markers =
uses_mock_api: tests that uses the mock API cluster.
uses_feast: tests that uses feast in some way
gemini_postcommit: gemini postcommits that need additional deps.
+ require_non_self_hosted: tests which can't be run on Beam's self hosted GitHub Actions runners. Usually this is because of issues with docker-in-docker or other environmental challenges. Context: https://github.com/apache/beam/pull/35585
Review Comment:
Thanks updated – also udpated the gradle task from `nonSelfHostedIntegrationTest` to `dockerInDockerIntegrationTest` to mirror this change
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #35585:
URL: https://github.com/apache/beam/pull/35585
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35585:
URL: https://github.com/apache/beam/pull/35585#discussion_r2205685193
##########
sdks/python/pytest.ini:
##########
@@ -70,6 +70,7 @@ markers =
uses_mock_api: tests that uses the mock API cluster.
uses_feast: tests that uses feast in some way
gemini_postcommit: gemini postcommits that need additional deps.
+ require_non_self_hosted: tests which can't be run on Beam's self hosted GitHub Actions runners. Usually this is because of issues with docker-in-docker or other environmental challenges. Context: https://github.com/apache/beam/pull/35585
Review Comment:
`require_no_docker_in_docker` SGTM - its not totally descriptive since I think this could apply in other cases, but I agree with the general idea here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35585:
URL: https://github.com/apache/beam/pull/35585#issuecomment-3070505862
Assigning reviewers:
R: @liferoad for label python.
R: @damccorm for label build.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #35585:
URL: https://github.com/apache/beam/pull/35585#discussion_r2205562207
##########
.github/workflows/beam_PostCommit_Python.yml:
##########
@@ -54,14 +54,15 @@ env:
jobs:
beam_PostCommit_Python:
name: ${{ matrix.job_name }} (${{ matrix.job_phrase }} ${{ matrix.python_version }})
- runs-on: [self-hosted, ubuntu-20.04, highmem22]
+ runs-on: ${{ matrix.os }}
timeout-minutes: 240
strategy:
fail-fast: false
matrix:
job_name: [beam_PostCommit_Python]
job_phrase: [Run Python PostCommit]
python_version: ['3.9', '3.10', '3.11', '3.12']
+ os: [[self-hosted, ubuntu-20.04, highmem22], [ubuntu-latest]]
Review Comment:
Could you add a small comment that describes why we're doing this here and in the other workflow?
##########
sdks/python/pytest.ini:
##########
@@ -70,6 +70,7 @@ markers =
uses_mock_api: tests that uses the mock API cluster.
uses_feast: tests that uses feast in some way
gemini_postcommit: gemini postcommits that need additional deps.
+ require_non_self_hosted: tests requiring specialized environments or infrastructure not present in self-hosted setups.
Review Comment:
```suggestion
require_non_self_hosted: tests which can't be run on Beam's self hosted GitHub Actions runners. Usually this is because of issues with docker-in-docker or other environmental challenges. Context: https://github.com/apache/beam/pull/35585
```
It would help to have more info here, feel free to use different wording though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] workflows+gradle: capture and run ML tests that require non self-hosted environment on `ubuntu-latest` [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #35585:
URL: https://github.com/apache/beam/pull/35585#discussion_r2205671469
##########
sdks/python/pytest.ini:
##########
@@ -70,6 +70,7 @@ markers =
uses_mock_api: tests that uses the mock API cluster.
uses_feast: tests that uses feast in some way
gemini_postcommit: gemini postcommits that need additional deps.
+ require_non_self_hosted: tests which can't be run on Beam's self hosted GitHub Actions runners. Usually this is because of issues with docker-in-docker or other environmental challenges. Context: https://github.com/apache/beam/pull/35585
Review Comment:
we try to decouple GitHub Actions with the other part of Beam code, so developers can run tests on their machines. Can we rename "require_non_self_hosted" to something like require_docker_in_docker or other informative marker names that does not assume the tests run in GitHub Action ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org