You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2024/07/29 18:52:28 UTC
[PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
lostluck opened a new pull request, #32012:
URL: https://github.com/apache/beam/pull/32012
Prism wasn't length prefix wrapping unknown coders for bundle internal PCollections, or for the parallel input pcollection. This prevented Java and Python SplittableDoFns from correctly encoding their sub element split results, since they pull their coder from the input PCollection.
Go is arguably incorrectly pulling it's coder from the DataSource transform itself, but explains why prism developed this bug.
This change ensures all the internal PCollections are wrapped, and updated with the correct coder. Further prism now only sends precisely the PCollections needed or used in the bundle, instead of the entire set of pipeline PCollections.
Fixes #32004.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32012:
URL: https://github.com/apache/beam/pull/32012#issuecomment-2257243576
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas merged PR #32012:
URL: https://github.com/apache/beam/pull/32012
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #32012:
URL: https://github.com/apache/beam/pull/32012#issuecomment-2257242376
R: @damondouglas
Definitely weird. Please review, but let's give it a day to get clean reruns I think.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32012:
URL: https://github.com/apache/beam/pull/32012#issuecomment-2256749751
Checks are failing. Will not request review until checks are succeeding. If you'd like to override that behavior, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [#32004] Ensure all pcollection coders are length prefixed if necessary. [beam]
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #32012:
URL: https://github.com/apache/beam/pull/32012#issuecomment-2259331154
Friendly ping.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org