You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "LEEKYE (via GitHub)" <gi...@apache.org> on 2025/06/16 01:31:12 UTC
[PR] Python dataflow option [beam]
LEEKYE opened a new pull request, #35298:
URL: https://github.com/apache/beam/pull/35298
**Please** add a meaningful description for your change here
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2257789707
##########
sdks/python/apache_beam/runners/worker/sdk_worker_main.py:
##########
@@ -167,7 +167,8 @@ def create_harness(environment, dry_run=False):
enable_heap_dump=enable_heap_dump,
data_sampler=data_sampler,
deferred_exception=deferred_exception,
- runner_capabilities=runner_capabilities)
+ runner_capabilities=runner_capabilities,
+ sdk_options=sdk_pipeline_options)
Review Comment:
Modified
##########
sdks/python/apache_beam/runners/worker/worker_status.py:
##########
@@ -250,6 +263,13 @@ def _log_lull_in_bundle_processor(self, bundle_process_cache):
if processor:
info = processor.state_sampler.get_info()
self._log_lull_sampler_info(info, instruction)
+ if self._restart_lull_timeout_ns and self._restart_lull(info):
+ sys.exit(1)
Review Comment:
Modified
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2151002410
##########
sdks/python/apache_beam/runners/worker/worker_status.py:
##########
@@ -250,6 +263,13 @@ def _log_lull_in_bundle_processor(self, bundle_process_cache):
if processor:
info = processor.state_sampler.get_info()
self._log_lull_sampler_info(info, instruction)
+ if self._restart_lull_timeout_ns and self._restart_lull(info):
+ sys.exit(1)
Review Comment:
On the design... we want to be very careful about adding `exit` calls (here and in Java). Because eventually code ends up getting used in unexpected contexts. It should be written without assumptions about the context it is called in, to the extent this is possible. So this could cause some other way of executing to exit unexpectedly. Can we raise an exception that bubbles up and causes the top level `main` to exit? That is really the only level where we should make any assumptions about being in charge of the process lifetime.
##########
sdks/python/apache_beam/runners/worker/worker_status.py:
##########
@@ -47,6 +48,8 @@
# 5 minutes * 60 seconds * 1000 millis * 1000 micros * 1000 nanoseconds
DEFAULT_LOG_LULL_TIMEOUT_NS = 5 * 60 * 1000 * 1000 * 1000
+DEFAULT_RESTART_LULL_TIMEOUT_NS = 10 * 60 * 1000 * 1000 * 1000
Review Comment:
This is actually an "exit/kill" timeout.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on PR #35298:
URL: https://github.com/apache/beam/pull/35298#issuecomment-2976719179
R: @apanich
R: @Abacn
R: @ryan-mbuashundip
R: @scwhittle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on PR #35298:
URL: https://github.com/apache/beam/pull/35298#issuecomment-2977998101
Can we log this timeout event clearly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "ryan-mbuashundip (via GitHub)" <gi...@apache.org>.
ryan-mbuashundip commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2167577148
##########
sdks/python/apache_beam/runners/worker/sdk_worker.py:
##########
@@ -176,7 +176,7 @@ def __init__(
# that should be reported to the runner when proocessing the first bundle.
deferred_exception=None, # type: Optional[Exception]
runner_capabilities=frozenset(), # type: FrozenSet[str]
- ):
+ sdk_options=None):
Review Comment:
```suggestion
sdk_options=None, # type: PipelineOptions
):
```
##########
sdks/python/apache_beam/runners/worker/sdk_worker.py:
##########
@@ -230,7 +231,8 @@ def default_factory(id):
status_address,
self._bundle_processor_cache,
self._state_cache,
- enable_heap_dump) # type: Optional[FnApiWorkerStatusHandler]
+ enable_heap_dump,
Review Comment:
```suggestion
enable_heap_dump, # type: Optional[FnApiWorkerStatusHandler]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE closed pull request #35298: Python dataflow option
URL: https://github.com/apache/beam/pull/35298
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "ryan-mbuashundip (via GitHub)" <gi...@apache.org>.
ryan-mbuashundip commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2152569472
##########
sdks/python/apache_beam/runners/worker/sdk_worker_main.py:
##########
@@ -167,7 +167,8 @@ def create_harness(environment, dry_run=False):
enable_heap_dump=enable_heap_dump,
data_sampler=data_sampler,
deferred_exception=deferred_exception,
- runner_capabilities=runner_capabilities)
+ runner_capabilities=runner_capabilities,
+ sdk_options=sdk_pipeline_options)
Review Comment:
This is probably best left as a TODO for now, but now that the `sdk_pipeline_options` is being passed in its entirety to the `SdkHarness` it might be a good idea to take out the other input parameters to the constructor which are derived from `sdk_pipeline_options` and define them in the constructor body.
##########
sdks/python/apache_beam/runners/worker/worker_status.py:
##########
@@ -250,6 +263,13 @@ def _log_lull_in_bundle_processor(self, bundle_process_cache):
if processor:
info = processor.state_sampler.get_info()
self._log_lull_sampler_info(info, instruction)
+ if self._restart_lull_timeout_ns and self._restart_lull(info):
+ sys.exit(1)
Review Comment:
+1, the exceptions route aligns well with how the timeout was implemented in the Java SDK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35298:
URL: https://github.com/apache/beam/pull/35298#issuecomment-2975049279
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #35298:
URL: https://github.com/apache/beam/pull/35298#issuecomment-2976723036
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2257790591
##########
sdks/python/apache_beam/runners/worker/sdk_worker.py:
##########
@@ -176,7 +176,7 @@ def __init__(
# that should be reported to the runner when proocessing the first bundle.
deferred_exception=None, # type: Optional[Exception]
runner_capabilities=frozenset(), # type: FrozenSet[str]
- ):
+ sdk_options=None):
Review Comment:
UPDATED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Python dataflow option [beam]
Posted by "LEEKYE (via GitHub)" <gi...@apache.org>.
LEEKYE commented on code in PR #35298:
URL: https://github.com/apache/beam/pull/35298#discussion_r2257791290
##########
sdks/python/apache_beam/runners/worker/sdk_worker.py:
##########
@@ -230,7 +231,8 @@ def default_factory(id):
status_address,
self._bundle_processor_cache,
self._state_cache,
- enable_heap_dump) # type: Optional[FnApiWorkerStatusHandler]
+ enable_heap_dump,
Review Comment:
UPDATED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org