You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "scwhittle (via GitHub)" <gi...@apache.org> on 2024/09/03 11:19:15 UTC
[I] [Bug]: Java generated RowCoder doesn't use encodingPositions when encoding/decoding the null set [beam]
scwhittle opened a new issue, #32388:
URL: https://github.com/apache/beam/issues/32388
### What happened?
Runners can ensure compatibility across multiple jobs runs of encoded schemas via encoding positions. Dataflow runner in particular sets encoding position overrides to maintain compatibility across pipelines.
[RowCoderGenerator](https://github.com/apache/beam/blob/64b4aa6fdde6a65b8157578036374623a47c18c2/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java#L139) utilizes these to map encoded index to the current Row representation. However it neglects to do so for writing the null bit set [here](https://github.com/apache/beam/blob/64b4aa6fdde6a65b8157578036374623a47c18c2/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java#L355) and when reading the null bit set [here](https://github.com/apache/beam/blob/64b4aa6fdde6a65b8157578036374623a47c18c2/sdks/java/core/src/main/java/org/apache/beam/sdk/coders/RowCoderGenerator.java#L428)
This leads to corruption when reading with encodedPositionOverrides that do not match the field ordering and additionally corrupts state when writing the null bit set based upon field ordering instead of encoding position.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: Java generated RowCoder doesn't use encodingPositions when encoding/decoding the null set [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle closed issue #32388: [Bug]: Java generated RowCoder doesn't use encodingPositions when encoding/decoding the null set
URL: https://github.com/apache/beam/issues/32388
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org