You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jrmccluskey (via GitHub)" <gi...@apache.org> on 2023/10/27 18:09:23 UTC
[PR] [WIP] Implement Cross-Bundle BatchElements [beam]
jrmccluskey opened a new pull request, #29175:
URL: https://github.com/apache/beam/pull/29175
Implements a stateful version of `BatchElements` that works across bundles, allowing for streaming pipelines that can batch elements in a dynamic fashion.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378105609
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -728,6 +836,86 @@ def expand(self, pcoll):
self._batch_size_estimator, self._element_size_fn))
+@typehints.with_input_types(T)
+@typehints.with_output_types(List[T])
+class StatefulBatchElements(PTransform):
+ """A Transform that batches elements for amortized processing.
+
+ This transform is designed to precede operations whose processing cost
+ is of the form
+
+ time = fixed_cost + num_elements * per_element_cost
+
+ where the per element cost is (often significantly) smaller than the fixed
+ cost and could be amortized over multiple elements. It consumes a PCollection
+ of element type T and produces a PCollection of element type List[T].
+
+ This transform attempts to find the best batch size between the minimim
+ and maximum parameters by profiling the time taken by (fused) downstream
+ operations. For a fixed batch size, set the min and max to be equal.
+
+ Elements are batched per-window and batches emitted in the window
+ corresponding to its contents. Each batch is emitted with a timestamp at
+ the end of their window.
+
+ Args:
+ min_batch_size: (optional) the smallest size of a batch
+ max_batch_size: (optional) the largest size of a batch
+ target_batch_overhead: (optional) a target for fixed_cost / time,
+ as used in the formula above
+ target_batch_duration_secs: (optional) a target for total time per bundle,
+ in seconds, excluding fixed cost
+ target_batch_duration_secs_including_fixed_cost: (optional) a target for
+ total time per bundle, in seconds, including fixed cost
+ element_size_fn: (optional) A mapping of an element to its contribution to
+ batch size, defaulting to every element having size 1. When provided,
+ attempts to provide batches of optimal total size which may consist of
+ a varying number of elements.
+ variance: (optional) the permitted (relative) amount of deviation from the
+ (estimated) ideal batch size used to produce a wider base for
+ linear interpolation
+ clock: (optional) an alternative to time.time for measuring the cost of
+ donwstream operations (mostly for testing)
+ record_metrics: (optional) whether or not to record beam metrics on
+ distributions of the batch size. Defaults to True.
+ """
+ def __init__(
+ self,
+ min_batch_size=1,
+ max_batch_size=10000,
+ target_batch_overhead=.05,
+ target_batch_duration_secs=10,
+ target_batch_duration_secs_including_fixed_cost=None,
+ max_batch_duration_secs=100,
Review Comment:
I take it back, testing through a branch in `BatchElements` is extremely easy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378066824
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -728,6 +836,86 @@ def expand(self, pcoll):
self._batch_size_estimator, self._element_size_fn))
+@typehints.with_input_types(T)
+@typehints.with_output_types(List[T])
+class StatefulBatchElements(PTransform):
+ """A Transform that batches elements for amortized processing.
+
+ This transform is designed to precede operations whose processing cost
+ is of the form
+
+ time = fixed_cost + num_elements * per_element_cost
+
+ where the per element cost is (often significantly) smaller than the fixed
+ cost and could be amortized over multiple elements. It consumes a PCollection
+ of element type T and produces a PCollection of element type List[T].
+
+ This transform attempts to find the best batch size between the minimim
+ and maximum parameters by profiling the time taken by (fused) downstream
+ operations. For a fixed batch size, set the min and max to be equal.
+
+ Elements are batched per-window and batches emitted in the window
+ corresponding to its contents. Each batch is emitted with a timestamp at
+ the end of their window.
+
+ Args:
+ min_batch_size: (optional) the smallest size of a batch
+ max_batch_size: (optional) the largest size of a batch
+ target_batch_overhead: (optional) a target for fixed_cost / time,
+ as used in the formula above
+ target_batch_duration_secs: (optional) a target for total time per bundle,
+ in seconds, excluding fixed cost
+ target_batch_duration_secs_including_fixed_cost: (optional) a target for
+ total time per bundle, in seconds, including fixed cost
+ element_size_fn: (optional) A mapping of an element to its contribution to
+ batch size, defaulting to every element having size 1. When provided,
+ attempts to provide batches of optimal total size which may consist of
+ a varying number of elements.
+ variance: (optional) the permitted (relative) amount of deviation from the
+ (estimated) ideal batch size used to produce a wider base for
+ linear interpolation
+ clock: (optional) an alternative to time.time for measuring the cost of
+ donwstream operations (mostly for testing)
+ record_metrics: (optional) whether or not to record beam metrics on
+ distributions of the batch size. Defaults to True.
+ """
+ def __init__(
+ self,
+ min_batch_size=1,
+ max_batch_size=10000,
+ target_batch_overhead=.05,
+ target_batch_duration_secs=10,
+ target_batch_duration_secs_including_fixed_cost=None,
+ max_batch_duration_secs=100,
Review Comment:
So to be clear, `max_batch_duration_secs` is the only additional parameter here, right? With that in mind, my vote would probably still be to make this part of `BatchElements`. I think we could drop in identical code, with the only difference being what we do in expand. One option would be to only use this implementation if `max_batch_duration_secs` is set (and make it default to None instead of 100). That would also give us the option to flip it to the default if we find that it performs as well/better later on (especially if we can detect that we're in a streaming pipeline).
That allows users to avoid needing to pick between 2 seemingly identical batching algorithms.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #29175:
URL: https://github.com/apache/beam/pull/29175
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378090494
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -728,6 +836,86 @@ def expand(self, pcoll):
self._batch_size_estimator, self._element_size_fn))
+@typehints.with_input_types(T)
+@typehints.with_output_types(List[T])
+class StatefulBatchElements(PTransform):
+ """A Transform that batches elements for amortized processing.
+
+ This transform is designed to precede operations whose processing cost
+ is of the form
+
+ time = fixed_cost + num_elements * per_element_cost
+
+ where the per element cost is (often significantly) smaller than the fixed
+ cost and could be amortized over multiple elements. It consumes a PCollection
+ of element type T and produces a PCollection of element type List[T].
+
+ This transform attempts to find the best batch size between the minimim
+ and maximum parameters by profiling the time taken by (fused) downstream
+ operations. For a fixed batch size, set the min and max to be equal.
+
+ Elements are batched per-window and batches emitted in the window
+ corresponding to its contents. Each batch is emitted with a timestamp at
+ the end of their window.
+
+ Args:
+ min_batch_size: (optional) the smallest size of a batch
+ max_batch_size: (optional) the largest size of a batch
+ target_batch_overhead: (optional) a target for fixed_cost / time,
+ as used in the formula above
+ target_batch_duration_secs: (optional) a target for total time per bundle,
+ in seconds, excluding fixed cost
+ target_batch_duration_secs_including_fixed_cost: (optional) a target for
+ total time per bundle, in seconds, including fixed cost
+ element_size_fn: (optional) A mapping of an element to its contribution to
+ batch size, defaulting to every element having size 1. When provided,
+ attempts to provide batches of optimal total size which may consist of
+ a varying number of elements.
+ variance: (optional) the permitted (relative) amount of deviation from the
+ (estimated) ideal batch size used to produce a wider base for
+ linear interpolation
+ clock: (optional) an alternative to time.time for measuring the cost of
+ donwstream operations (mostly for testing)
+ record_metrics: (optional) whether or not to record beam metrics on
+ distributions of the batch size. Defaults to True.
+ """
+ def __init__(
+ self,
+ min_batch_size=1,
+ max_batch_size=10000,
+ target_batch_overhead=.05,
+ target_batch_duration_secs=10,
+ target_batch_duration_secs_including_fixed_cost=None,
+ max_batch_duration_secs=100,
Review Comment:
That is correct, it's the only additional parameter. The separate transform makes things easier for testing IMO, but from an actual implementation standpoint it isn't particularly difficult to manage. I'm not opposed to rerouting BatchElements if the param is set, since that satisfies the "optional route" rollout step we discussed offline
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378198383
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -646,6 +647,113 @@ def finish_bundle(self):
self._target_batch_size = self._batch_size_estimator.next_batch_size()
+def _pardo_stateful_batch_elements(
+ input_coder: coders.Coder,
+ batch_size_estimator: _BatchSizeEstimator,
+ max_buffering_duration_secs: int,
+ clock=time.time):
+ ELEMENT_STATE = BagStateSpec('values', input_coder)
+ COUNT_STATE = CombiningValueStateSpec('count', input_coder, CountCombineFn())
+ BATCH_SIZE_STATE = ReadModifyWriteStateSpec('batch_size', input_coder)
+ WINDOW_TIMER = TimerSpec('window_end', TimeDomain.WATERMARK)
+ BUFFERING_TIMER = TimerSpec('buffering_end', TimeDomain.REAL_TIME)
+ BATCH_ESTIMATOR_STATE = ReadModifyWriteStateSpec(
+ 'batch_estimator', coders.PickleCoder())
+
+ class _StatefulBatchElementsDoFn(DoFn):
+ def process(
+ self,
+ element,
+ window=DoFn.WindowParam,
+ element_state=DoFn.StateParam(ELEMENT_STATE),
+ count_state=DoFn.StateParam(COUNT_STATE),
+ batch_size_state=DoFn.StateParam(BATCH_SIZE_STATE),
+ batch_estimator_state=DoFn.StateParam(BATCH_ESTIMATOR_STATE),
+ window_timer=DoFn.TimerParam(WINDOW_TIMER),
+ buffering_timer=DoFn.TimerParam(BUFFERING_TIMER)):
+ state_estimator = batch_estimator_state.read()
+ if state_estimator is not None:
+ batch_estimator = state_estimator
+ else:
+ # Should only happen on the first element
+ batch_estimator = batch_size_estimator
Review Comment:
Can we move this read block into the `if target_size is None` block? That way we're only doing the state read if we need it
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -712,10 +824,20 @@ def __init__(
clock=clock,
record_metrics=record_metrics)
self._element_size_fn = element_size_fn
+ self._max_batch_dur = max_batch_duration_secs
+ self._clock = clock
def expand(self, pcoll):
if getattr(pcoll.pipeline.runner, 'is_streaming', False):
raise NotImplementedError("Requires stateful processing (BEAM-2687)")
+ elif self._max_batch_dur is not None:
+ coder = coders.registry.get_coder(pcoll)
+ return pcoll | WithKeys(0) | ParDo(
Review Comment:
Non-blocking for this PR, but something we may want to consider; rather than using a single fixed key, does it make sense to try to have a single key per worker somehow? (one way to do this would be using multi_process_shared.py)
That way we're still batching per machine in a parallelizable way, but we get stateful batching across bundles.
The current implementation is likely still useful for many use cases where batching is not the expensive part (e.g. RunInference) or there are few workers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1785849688
Run Python_Examples PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378855433
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -646,6 +647,113 @@ def finish_bundle(self):
self._target_batch_size = self._batch_size_estimator.next_batch_size()
+def _pardo_stateful_batch_elements(
+ input_coder: coders.Coder,
+ batch_size_estimator: _BatchSizeEstimator,
+ max_buffering_duration_secs: int,
+ clock=time.time):
+ ELEMENT_STATE = BagStateSpec('values', input_coder)
+ COUNT_STATE = CombiningValueStateSpec('count', input_coder, CountCombineFn())
+ BATCH_SIZE_STATE = ReadModifyWriteStateSpec('batch_size', input_coder)
+ WINDOW_TIMER = TimerSpec('window_end', TimeDomain.WATERMARK)
+ BUFFERING_TIMER = TimerSpec('buffering_end', TimeDomain.REAL_TIME)
+ BATCH_ESTIMATOR_STATE = ReadModifyWriteStateSpec(
+ 'batch_estimator', coders.PickleCoder())
+
+ class _StatefulBatchElementsDoFn(DoFn):
+ def process(
+ self,
+ element,
+ window=DoFn.WindowParam,
+ element_state=DoFn.StateParam(ELEMENT_STATE),
+ count_state=DoFn.StateParam(COUNT_STATE),
+ batch_size_state=DoFn.StateParam(BATCH_SIZE_STATE),
+ batch_estimator_state=DoFn.StateParam(BATCH_ESTIMATOR_STATE),
+ window_timer=DoFn.TimerParam(WINDOW_TIMER),
+ buffering_timer=DoFn.TimerParam(BUFFERING_TIMER)):
+ state_estimator = batch_estimator_state.read()
+ if state_estimator is not None:
+ batch_estimator = state_estimator
+ else:
+ # Should only happen on the first element
+ batch_estimator = batch_size_estimator
Review Comment:
Done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1787773019
R: @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378888012
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -712,10 +824,20 @@ def __init__(
clock=clock,
record_metrics=record_metrics)
self._element_size_fn = element_size_fn
+ self._max_batch_dur = max_batch_duration_secs
+ self._clock = clock
def expand(self, pcoll):
if getattr(pcoll.pipeline.runner, 'is_streaming', False):
raise NotImplementedError("Requires stateful processing (BEAM-2687)")
+ elif self._max_batch_dur is not None:
+ coder = coders.registry.get_coder(pcoll)
+ return pcoll | WithKeys(0) | ParDo(
Review Comment:
Oh I missed that (or forgot), hopefully I wasn't against it initially?
Regardless, I like having this as our first pass, we can see how it performs and go from there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1785849219
Run Python_Coverage PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1787776278
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on code in PR #29175:
URL: https://github.com/apache/beam/pull/29175#discussion_r1378856924
##########
sdks/python/apache_beam/transforms/util.py:
##########
@@ -712,10 +824,20 @@ def __init__(
clock=clock,
record_metrics=record_metrics)
self._element_size_fn = element_size_fn
+ self._max_batch_dur = max_batch_duration_secs
+ self._clock = clock
def expand(self, pcoll):
if getattr(pcoll.pipeline.runner, 'is_streaming', False):
raise NotImplementedError("Requires stateful processing (BEAM-2687)")
+ elif self._max_batch_dur is not None:
+ coder = coders.registry.get_coder(pcoll)
+ return pcoll | WithKeys(0) | ParDo(
Review Comment:
At one point I suggested keying with worker IDs, it may be worth coming back to that idea
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1785849902
Run Python_Transforms PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1785849420
Run Python_Dataframes PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [WIP] Implement Cross-Bundle BatchElements [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1785883170
We're currently processing your upload. This comment will be updated when the results are available.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Implement Cross-Bundle BatchElements [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on PR #29175:
URL: https://github.com/apache/beam/pull/29175#issuecomment-1787912265
Run Python_Coverage PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org