You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2024/07/30 22:49:07 UTC

[PR] [#32003][prism] Support empty transform input sets, such as for flattens. [beam]

lostluck opened a new pull request, #32029:
URL: https://github.com/apache/beam/pull/32029

   In very rare cases a transform, such as a flatten can be in a pipeline without any real inputs.
   
   Without inputs, nothing kicked off watermark processing, which then blocked use of the empty flatten output as a side input, which blocked that transform from executing.
   
   This manifested as the upstream stage (the flatten), never getting watermark refreshed, and therefore the downstream stage never being able to execute, since the side input data from the upstream stage was never ready.
   
   The solution in this PR is when a stage is added without inputs, it must be part of the initial watermark refresh set. Further taken to keep this part of the "impulse" set when a TestStream is involved in the Pipeline, since that manipulates initial refreshes.
   
   Additionally adds a line to the stuck pipeline dump to clarify the inputs and output consumers of each stage, to make the progression structure clearer.
   
   Fixes #32003.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [ ] Update `CHANGES.md` with noteworthy changes.
    - [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/workflows/Build%20python%20source%20distribution%20and%20wheels/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/workflows/Python%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/workflows/Java%20Tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/workflows/Go%20tests/badge.svg?branch=master&event=schedule)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [#32003][prism] Support empty transform input sets, such as for flattens. [beam]

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas merged PR #32029:
URL: https://github.com/apache/beam/pull/32029


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [#32003][prism] Support empty transform input sets, such as for flattens. [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #32029:
URL: https://github.com/apache/beam/pull/32029#issuecomment-2259370008

   Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @jrmccluskey for label go.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [#32003][prism] Support empty transform input sets, such as for flattens. [beam]

Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on code in PR #32029:
URL: https://github.com/apache/beam/pull/32029#discussion_r1697730564


##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -211,6 +211,16 @@ func (em *ElementManager) AddStage(ID string, inputIDs, outputIDs []string, side
 	for _, input := range inputIDs {
 		em.consumers[input] = append(em.consumers[input], ss.ID)
 	}
+
+	// In very rare cases, we can have a stage without any inputs, such as a flatten.
+	// In that case, there's nothing that will start the watermark refresh cycle,
+	// so we must do it here.
+	if len(inputIDs) == 0 {

Review Comment:
   Effectively yes, but basically it's due to flatten being special: all Other Stages must have a single parallel input pCollection. Flattens may have any number, which apparently includes 0.
   
   In this case, by construction, the flatten has no inputs to begin with.
   
   Impulses and TestStream are also special: Impulses are required to kick off SDK side processing. TestStream is just weird.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [#32003][prism] Support empty transform input sets, such as for flattens. [beam]

Posted by "damondouglas (via GitHub)" <gi...@apache.org>.
damondouglas commented on code in PR #32029:
URL: https://github.com/apache/beam/pull/32029#discussion_r1697715863


##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -211,6 +211,16 @@ func (em *ElementManager) AddStage(ID string, inputIDs, outputIDs []string, side
 	for _, input := range inputIDs {
 		em.consumers[input] = append(em.consumers[input], ss.ID)
 	}
+
+	// In very rare cases, we can have a stage without any inputs, such as a flatten.
+	// In that case, there's nothing that will start the watermark refresh cycle,
+	// so we must do it here.
+	if len(inputIDs) == 0 {

Review Comment:
   Non-PR blocking comment to validate my understanding. Do I understand correctly?
   
   In the PTransform with no PCollection case, there are no inputIDs because the result of `internal.transformPreparer.PrepareTransform` returns an `internal.prepareResult` where `prepResult.SubbedComps.GetPcollections()` is empty.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org