You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2021/08/24 21:34:06 UTC
[GitHub] [beam] reuvenlax opened a new pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
reuvenlax opened a new pull request #15382:
URL: https://github.com/apache/beam/pull/15382
In order to implement BEAM-12795, we need to add non-trivial expansion to ParDo.MultiOuput. This cannot be done today since runners treat MultiOutput as a primitive and replace it wholesale. In this PR we create a new ParDo.MultiOutputPrimitive and update runners to replace that instead.
All runners in the Beam codebase have been updated. Any runners outside the Beam codebase will need to start replacing MultiOutputPrimitive before updating to the next Beam release.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #15382:
URL: https://github.com/apache/beam/pull/15382#issuecomment-904992283
R: @lukecwik
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #15382:
URL: https://github.com/apache/beam/pull/15382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] closed pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
github-actions[bot] closed pull request #15382:
URL: https://github.com/apache/beam/pull/15382
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15382:
URL: https://github.com/apache/beam/pull/15382#issuecomment-1030616247
This pull request has been marked as stale due to 60 days of inactivity. It will be closed in 1 week if no further activity occurs. If you think that’s incorrect or this pull request requires a review, please simply write any comment. If closed, you can revive the PR at any time and @mention a reviewer or discuss it on the dev@beam.apache.org list. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] reuvenlax commented on pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
reuvenlax commented on pull request #15382:
URL: https://github.com/apache/beam/pull/15382#issuecomment-907484209
I was wondering about that, but Dataflow at least replaces things in runner
expansion so I'm not sure if the update would fail or not. I could do
this only for the case of schema key expansion, but that requires updating
every runner to handle both primitive names, which would be much trickier
to do.
Do we have any. update tests in Beam I could run?
On Fri, Aug 27, 2021 at 2:18 PM Lukasz Cwik ***@***.***>
wrote:
> ***@***.**** commented on this pull request.
>
> Unfortunately I didn't think of this before but won't this break pipeline
> update for everyone since every ParDo's name changed (unless they provide
> the rename map which seems painful)?
> ------------------------------
>
> In sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
> <https://github.com/apache/beam/pull/15382#discussion_r697719882>:
>
> > @@ -950,13 +951,13 @@ public PCollectionTuple expand(PCollection<? extends InputT> input) {
> // Need to figure out where to validate this.
>
> Is there a reason why you didn't move the rest of this logic into
> MultiOutputPrimitive?
>
> —
> You are receiving this because you authored the thread.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/beam/pull/15382#pullrequestreview-740888320>,
> or unsubscribe
> <https://github.com/notifications/unsubscribe-auth/AFAYJVLUYWNN55Y3WPMDF4TT676JRANCNFSM5CXWLPKA>
> .
> Triage notifications on the go with GitHub Mobile for iOS
> <https://apps.apple.com/app/apple-store/id1477376905?ct=notification-email&mt=8&pt=524675>
> or Android
> <https://play.google.com/store/apps/details?id=com.github.android&referrer=utm_campaign%3Dnotification-email%26utm_medium%3Demail%26utm_source%3Dgithub>.
>
>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15382:
URL: https://github.com/apache/beam/pull/15382#issuecomment-1037213778
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lukecwik commented on a change in pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
lukecwik commented on a change in pull request #15382:
URL: https://github.com/apache/beam/pull/15382#discussion_r697719882
##########
File path: sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ParDo.java
##########
@@ -950,13 +951,13 @@ public PCollectionTuple expand(PCollection<? extends InputT> input) {
// Need to figure out where to validate this.
Review comment:
Is there a reason why you didn't move the rest of this logic into MultiOutputPrimitive?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #15382: [BEAM-12795] In preparation for implementing BEAM-12795, create a new ParDo primitive
Posted by GitBox <gi...@apache.org>.
github-actions[bot] commented on pull request #15382:
URL: https://github.com/apache/beam/pull/15382#issuecomment-1037213778
This pull request has been closed due to lack of activity. If you think that is incorrect, or the pull request requires review, you can revive the PR at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org