You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/11/26 22:43:58 UTC
[GitHub] [beam] naddeoa opened a new issue, #24360: [Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
naddeoa opened a new issue, #24360:
URL: https://github.com/apache/beam/issues/24360
### What would you like to happen?
I'm using a tool in my beam template that benefits largely from batching. In general, the more the better. I noticed that my `DoFn` subclass's `process_batch` is only ever getting 4096 items. This is apparently [hard coded](https://github.com/apache/beam/blob/eed1713b5b8dc8e83ffb1fa43a4d0564083a23ce/sdks/python/apache_beam/runners/worker/operations.py#L262) and can't be changed.
Is there something special about 4096? If not, how difficult would it be to allow users to specify ideal batch sizes?
### Issue Priority
Priority: 3
### Issue Component
Component: sdk-py-core
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24360: [Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24360:
URL: https://github.com/apache/beam/issues/24360#issuecomment-1334593939
Looking at https://github.com/apache/beam/pull/17384 , the limit has been there since the very beginning, I suspect we needed to have some limit, the 4096 is a guesstimate. There may be an tradeoff with memory footprint, but I think increasing the limit would work. You can try to patch the SDK and supply a modified version with --sdk_location flag. s.apache.org/beam-python-dev-wiki has pointers on how to build an SDK tarball.
If you can cleanly configure it (e.g. via pipeline option), feel free to make a PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] kennknowles commented on issue #24360: [Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
Posted by GitBox <gi...@apache.org>.
kennknowles commented on issue #24360:
URL: https://github.com/apache/beam/issues/24360#issuecomment-1334564449
@tvalentyn @damccorm this probably falls in the intersection of your interests
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] naddeoa commented on issue #24360: [Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
Posted by GitBox <gi...@apache.org>.
naddeoa commented on issue #24360:
URL: https://github.com/apache/beam/issues/24360#issuecomment-1335928272
@tvalentyn thanks.
Some questions/assumptions:
1. At a high level, what would an ideal value for this look like? I assume it has to be some percentage of the memory on the machine type that you're using for workers. Do you know how much memory its safe to consume purely in data?
2. Does the batch size lead to any disk usage as well (not counting whatever someone's custom DoFn might be doing) or does it just imply more memory usage.
3. The best user experience would be to have the batch size determined dynamically so no one has to think about it. Would that happen through size estimates? That might be getting ahead of myself since beam would have to know a lot about the memory available on the workers as well.
4. Why was 4096 chosen? At a glance, if we're talking about running Dataflow on your average BigQuery data in a schema with a dozen rows or text/numbers, then 4096 might not even be a mb. Wouldn't a more aggressive default still have been a safe estimate?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on issue #24360: [Feature Request]: Customize MAX_BATCH_SIZE in DoFn.process_batch
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on issue #24360:
URL: https://github.com/apache/beam/issues/24360#issuecomment-1335958887
I don't think there is a disk usage overhead associated w/ larger batch.
I would be curious of how performance changes with increasing max batch size before doing an advanced logic to dynamically change the default.
Note that the upper bound for a batch size would be the bundle size (determined by the runner).
@robertwb do you know how does 4096 relate to typical bundle sizes in Dataflow?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org