You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "joar (via GitHub)" <gi...@apache.org> on 2023/10/03 09:16:53 UTC
[I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
joar opened a new issue, #28793:
URL: https://github.com/apache/beam/issues/28793
### What happened?
org.apache.beam.runners.direct.BoundedReadEvaluatorFactory.InputProvider#getInitialInputs
org.apache.beam.sdk.io.gcp.bigtable.BigtableIO.BigtableSource#split
Imagine a case where in:
https://github.com/apache/beam/blob/282d027f16214c9581b1d19073252425cff53ee9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java#L215-L217
- `targetParallelism` is `32`
- `source.getEstimatedByteSize()` is `10`
then
- `bytesPerBundle` will be `0`
so
https://github.com/apache/beam/blob/282d027f16214c9581b1d19073252425cff53ee9/runners/direct-java/src/main/java/org/apache/beam/runners/direct/BoundedReadEvaluatorFactory.java#L217
will be called with the values: `split.source(0L, options)`
In `OffsetBasedSource#split`, this desired-0-sized split is handled:
https://github.com/apache/beam/blob/282d027f16214c9581b1d19073252425cff53ee9/sdks/java/core/src/main/java/org/apache/beam/sdk/io/OffsetBasedSource.java#L115-L116
But `BigtableSource#split` does not seem to handle the desired-0-sized split:
https://github.com/apache/beam/blob/282d027f16214c9581b1d19073252425cff53ee9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1328-L1333
so a few frames down the road from `BigtableSource#split` you'll end up violating this `checkArgument`:
https://github.com/apache/beam/blob/71c8459633ec86e576eca080a26be9f42474ecb2/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java#L1623-L1626
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28793:
URL: https://github.com/apache/beam/issues/28793#issuecomment-1745183469
Thanks for reporting the issue. Interesting edge case, since it has been root caused, would you mind entering a PR for fix, thanks for the contribution!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
Posted by "joar (via GitHub)" <gi...@apache.org>.
joar commented on issue #28793:
URL: https://github.com/apache/beam/issues/28793#issuecomment-1745256245
The question is, should it be fixed in `BigtableSource`, `BoundedReadEvaluatorFactory` or both?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
Posted by "joar (via GitHub)" <gi...@apache.org>.
joar commented on issue #28793:
URL: https://github.com/apache/beam/issues/28793#issuecomment-1745246300
The scenario that makes this edge case pop up for us is running pipelines in CI against the Bigtable emulator. The key of course being tiny tables.
I'll enter a PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28793:
URL: https://github.com/apache/beam/issues/28793#issuecomment-1745361478
Fix BoundedReadEvaluatorFactory
- Pro: prevent errorprone on all implementation like this
- Con: changed the behavior of java core, the scope of impact is unknown
Fix BigtableSource
- Pro: safe
- Con: just fix one use case
Based on this understanding I would vote for Fix BigtableSource for now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0" [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #28793: [Bug]: BigtableSource "Desired bundle size 0 bytes must be greater than 0"
URL: https://github.com/apache/beam/issues/28793
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org