You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "aIbrahiim (via GitHub)" <gi...@apache.org> on 2026/03/28 01:10:50 UTC
[PR] increase timeout via posargs [beam]
aIbrahiim opened a new pull request, #37989:
URL: https://github.com/apache/beam/pull/37989
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #37989:
URL: https://github.com/apache/beam/pull/37989#issuecomment-4146269342
> [!NOTE]
> Gemini is unable to generate a summary for this pull request due to the file types involved not being currently supported.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017878956
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
sure we can try; it might be possible to also run that instrumentation on a separate fork if you can repro reliably on a fork and for whatever reason we don't want to merge the change (e.g. it makes tests much slower or unstable)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017372395
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
if we can repro this stuckness locally, or ssh into GHA worker that is stuck, then we could spy on the python process with tools like `pystack`, and then examine stracktraces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025192582
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
re: 1 --The issue ties to YAML external providers: they clone a venv in yaml_provider via clonevirtualenv so source is either a fresh mini venv, or in .dev the whole tox venv (path from base_python) and in CI that tree is live, so paths like tmp/ can disappear mid-copy, fits what we saw so I suggesst ignore tmp/, __pycache__/, .pytest_cache/, pip/build temps (or replace clonevirtualenv with a copy that supports ignore / only needed dirs) and for .dev/CI, prefer _create_venv_from_scratch (or a template venv) instead of cloning the full tox env, slower, stabler. Optional unit test with a fake venv + volatile tmp/ to guard regressions.
re: 2 -- subprocess_server waits on gRPC channel ready for expansion_service_main until pytest-timeout. faulthandler showed xdist/execnet sometimes, xdist off for one Python version didn’t fix other legs, parallelism isnt the whole story so I suggesst on child exit or long wait, surface exit code + stdout/stderr (and maybe fail faster with a clear error).
For ML YAML, preinstall/pin heavy deps in tox so slow pip/import/model startup doesn’t look like a hang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3024022664
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
I added faulthandler in process (periodic tracebacks + SIGUSR1) and tried pystack remote in the background with artifact uploads and what happened: pystack pretty much always failed, Operation not permitted / no CAP_SYS_PTRACE on the runner, so relying on attach from outside isn’t realistic there. faulthandler helped though, I saw xdist/execnet stuck in some dumps, so parallelism is part of the story but not everything and I also tried turning off xdist for one Python version, stuff still failed elsewhere while concrete examples: 3.11 hit clonevirtualenv / shutil copying the live tox venv (junk under tmp/ etc.), 3.12/3.13 sat in subprocess_server until pytest-timeout waiting for “grpc channel ready” on the Python expansion subprocess. so not one root bug for all flakes, but two places worth real fixes: (1) venv clone for YAML expansion and (2) expansion subprocess actually coming up + better errors when it doesn’t. xdist might still make it worse sometimes. @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029668552
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
Ahh Makes sense so I willll add a short comment above the _ci check explaining that in CI the dev venv we infer from base_python is often the live tox/sandbox env so clonevirtualenv can race on ephemeral paths (tmp/, caches, etc.); outside CI we keep the fast .dev clone path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3016361813
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
Review Comment:
Ahh I didn't see this run, then icreasing pytest timeoutcan mask hangs and extend stuck runs so i revert the timeout bump and keep this PR focused on grpc stability env vars only as a short term mitigation @Abacn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017077643
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
Review Comment:
SGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017374505
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
that could give some clues.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017646534
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
> so should I run with SSH access on a stuck self-hosted runner
is it an option? is it easy?
how often does this issue reproduce?
overall, i'd pick whatever option is more straightforward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025192582
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
re: 1 --The issue ties to YAML external providers: they clone a venv in yaml_provider via clonevirtualenv so source is either a fresh mini venv, or in .dev the whole tox venv (path from base_python) and in CI that tree is live, so paths like tmp/ can disappear mid-copy, fits what we saw so I suggesst ignore tmp/, __pycache__/, .pytest_cache/, pip/build temps (or replace clonevirtualenv with a copy that supports ignore / only needed dirs) and for .dev/CI, prefer _create_venv_from_scratch (or a template venv) instead of cloning the full tox env, slower, stabler. Optional unit test with a fake venv + volatile tmp/ to guard regressions.
re: 2 -- subprocess_server waits on gRPC channel ready for expansion_service_main until pytest-timeout. faulthandler showed xdist/execnet sometimes, xdist off for one Python version didn’t fix other legs, parallelism isnt the whole story so I suggesst on child exit or long wait, surface exit code + stdout/stderr (and maybe fail faster with a clear error).
For ML YAML, preinstall/pin heavy deps in tox so slow pip/import/model startup doesn’t look like a hang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn merged PR #37989:
URL: https://github.com/apache/beam/pull/37989
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017372395
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
if we can repro this stuckness locally, or ssh into GHA worker, we could spy on the python process with tools like `pystack`, and then examine stracktraces.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3016223008
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
Wondering if there is some racing condition for tests run in parllell cc: @tvalentyn. Nevertheless we can add these environment variables at the moment.
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
Review Comment:
Not sure further increasing timeout and/or rerun could help. We already see the test suite stuck for >5h e.g. https://github.com/apache/beam/actions/runs/23779768061
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "Amar3tto (via GitHub)" <gi...@apache.org>.
Amar3tto commented on PR #37989:
URL: https://github.com/apache/beam/pull/37989#issuecomment-4178180271
Failed checks are unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029521670
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
cc: @sergiitk FYI - we might be seeing stuckness issues in other, pure python test suites, where there are processes and subprocesses communicating over GRPC.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017777876
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
rom my side, SSH into a stuck self-hosted runner is possible probably only if we have runner access granted so it’s not always the easiest path to start with and locally I couldn’t reproduce yet due to WSL network limits (can’t reach plugins.gradle.org/pypi) so I dont have a solid repro rate from local runs so given that the most straightforward option is to add temporary CI instrumentation (faulthandler + periodic pystack dumps uploaded as artifacts) so every stuck run automatically gives us stack traces
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
from my side, SSH into a stuck self-hosted runner is possible probably only if we have runner access granted so it’s not always the easiest path to start with and locally I couldn’t reproduce yet due to WSL network limits (can’t reach plugins.gradle.org/pypi) so I dont have a solid repro rate from local runs so given that the most straightforward option is to add temporary CI instrumentation (faulthandler + periodic pystack dumps uploaded as artifacts) so every stuck run automatically gives us stack traces
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025203167
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
I will add the same env var to Python PreCommit and see if it changes the expansion / xdist behavior, will update you back whether it helps
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029668552
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
Ahh Makes sense so I will add a short comment above the _ci check explaining that in CI the dev venv we infer from base_python is often the live tox/sandbox env so clonevirtualenv can race on ephemeral paths (tmp/, caches, etc.); outside CI we keep the fast .dev clone path
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025151875
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
re: 1 -- is there something specific to YAML suite? do you have some suggestions (e.g. skip copying tmp?) I
re: 2 -- let's try to indentify the prob
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on PR #37989:
URL: https://github.com/apache/beam/pull/37989#issuecomment-4184257521
thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029568554
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -57,6 +57,7 @@ env:
PYTHONHASHSEED: "0"
OMP_NUM_THREADS: "1"
OPENBLAS_NUM_THREADS: "1"
+ GRPC_ENABLE_FORK_SUPPORT: "0"
Review Comment:
let's add a TODO to remove once https://github.com/grpc/grpc/issues/37710 is fixed, like we have in the other suite.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025151875
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
Good findings!
re: 1 -- is there something specific to YAML suite? do you have some suggestions (e.g. skip copying tmp?) I
re: 2 -- Could you try setting this environment variable and seeing if it helps: https://github.com/apache/beam/blob/ee02db885b83b5233341f63f1a2cde98dc0193df/.github/workflows/beam_PreCommit_GoPortable.yml#L42 .
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025151875
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
Good findings!
re: 1 -- is there something specific to YAML suite? do you have some suggestions (e.g. skip copying tmp?) I
re: 2 --
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3017587193
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
so should I run with SSH access on a stuck self-hosted runner and collect pystack traces there, or add temporary CI instrumentation (faulthandler + periodic pystack dumps as artifacts) so the next stuck run captures stack traces automatically? @tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #37989:
URL: https://github.com/apache/beam/pull/37989#issuecomment-4146325186
Assigning reviewers:
R: @Abacn for label build.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029521670
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
cc: @sergiitk FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3025199185
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
re: 1 --The issue ties to YAML external providers: they clone a venv in yaml_provider via clonevirtualenv so source is either a fresh mini venv, or in .dev the whole tox venv (path from base_python) and in CI that tree is live, so paths like tmp/ can disappear mid-copy, fits what we saw so I suggesst ignore tmp/, pycache/, .pytest_cache/, pip/build temps (or replace clonevirtualenv with a copy that supports ignore / only needed dirs) and for .dev/CI, prefer _create_venv_from_scratch (or a template venv) instead of cloning the full tox env, slower, stabler. Optional unit test with a fake venv + volatile tmp/ to guard regressions.
re: 2 -- subprocess_server waits on gRPC channel ready for expansion_service_main until pytest-timeout. faulthandler showed xdist/execnet sometimes, xdist off for one Python version didn’t fix other legs, parallelism isnt the whole story so I suggesst on child exit or long wait, surface exit code + stdout/stderr (and maybe fail faster with a clear error).
For ML YAML, preinstall/pin heavy deps in tox so slow pip/import/model startup doesn’t look like a hang
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3024028572
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
https://github.com/apache/beam/actions/runs/23839677633
https://github.com/apache/beam/actions/runs/23854292029
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3027046884
##########
.github/workflows/beam_PreCommit_Python.yml:
##########
@@ -111,12 +111,14 @@ jobs:
env:
TOX_TESTENV_PASSENV: "DOCKER_*,TESTCONTAINERS_*,TC_*,BEAM_*,GRPC_*,OMP_*,OPENBLAS_*,PYTHONHASHSEED,PYTEST_*"
# Aggressive retry and timeout settings for flaky CI
- PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=600 --disable-warnings"
+ PYTEST_ADDOPTS: "-v --tb=short --maxfail=5 --durations=30 --reruns=5 --reruns-delay=15 --timeout=900 --disable-warnings"
# Container stability - much more generous timeouts
TC_TIMEOUT: "300"
TC_MAX_TRIES: "15"
TC_SLEEP_TIME: "5"
# Additional gRPC stability for flaky environment
+ GRPC_ARG_KEEPALIVE_TIME_MS: "60000"
Review Comment:
@tvalentyn I added GRPC_ENABLE_FORK_SUPPORT: "0" in Python PreCommit and kept the YAML side stabilization in yaml_provider then I reran the workflow and it passed https://github.com/apache/beam/actions/runs/23892113364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029597734
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
can we add rationale here for this change? It would be better to make this change in a separate PR and clarify context + rationale in the description in addition to small the comment here.
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
can we add rationale here for this change? Also, it would be better to make this change in a separate PR and clarify context + rationale in the description in addition to small the comment here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3029607014
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
> if '.dev' in beam_version and not _ci:
I wonder if the `if '.dev'` specifically targeted the CI. In this case removing the whole branch makes more sense.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] increase timeout via posargs [beam]
Posted by "aIbrahiim (via GitHub)" <gi...@apache.org>.
aIbrahiim commented on code in PR #37989:
URL: https://github.com/apache/beam/pull/37989#discussion_r3032450652
##########
sdks/python/apache_beam/yaml/yaml_provider.py:
##########
@@ -1412,7 +1412,8 @@ def _create_venv_from_clone(
@classmethod
def _create_venv_to_clone(cls, base_python: str) -> str:
- if '.dev' in beam_version:
+ _ci = os.environ.get('CI', '').lower() in ('true', '1', 'yes')
Review Comment:
I split this out into its own PR https://github.com/apache/beam/pull/38063
@tvalentyn
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org