You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "jmdobry (via GitHub)" <gi...@apache.org> on 2023/11/08 19:02:58 UTC
[I] [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size [beam]
jmdobry opened a new issue, #29350:
URL: https://github.com/apache/beam/issues/29350
### What happened?
When using `BatchElements`, one can precisely control the size of each batch by setting `min_batch_size` and `max_batch_size` to the same value and by also providing `element_size_fn`. It almost works. There are cases where exceeding max_batch_size in any way is unacceptable (example, APIs that have a max request payload size).
However, the `process` function of `_GlobalWindowsBatchingDoFn` blindly adds elements to a batch _before_ checking to see if adding that element would exceed. This can be fixed by changing `process` to only add an element to a batch if the batch is empty OR adding the element will not exceed match_batch_size. Example fixed implementation:
```python
if self._running_batch_size and (
(self._running_batch_size + self._element_size_fn(element))
>= self._target_batch_size
):
with self._batch_size_estimator.record_time(self._running_batch_size):
yield window.GlobalWindows.windowed_value_at_end_of_window(self._batch)
self._batch = []
self._running_batch_size = 0
self._target_batch_size = self._batch_size_estimator.next_batch_size()
# End if
self._batch.append(element)
self._running_batch_size += self._element_size_fn(element)
```
### Issue Priority
Priority: 3 (minor)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #29350:
URL: https://github.com/apache/beam/issues/29350#issuecomment-1810685401
hi @jmdobry ! Thanks for reporting. Would you be interested in contributing a fix (and ideally a test) to address this ? Let me know if you have any questions. FYI we also have https://s.apache.org/beam-python-dev-wiki for development tips. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey commented on issue #29350:
URL: https://github.com/apache/beam/issues/29350#issuecomment-1804190540
The proposed change LGTM, feel free to open a PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size [beam]
Posted by "jrmccluskey (via GitHub)" <gi...@apache.org>.
jrmccluskey closed issue #29350: [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size
URL: https://github.com/apache/beam/issues/29350
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: apache_beam.transforms.util.BatchElements allows creating batches that exceed max_batch_size [beam]
Posted by "AnandInguva (via GitHub)" <gi...@apache.org>.
AnandInguva commented on issue #29350:
URL: https://github.com/apache/beam/issues/29350#issuecomment-1804084788
cc: @jrmccluskey who has some context on BatchElements
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org