[I] [prism] Support Empty Flattens as Side Input - no way to make progress with pending elements [beam]

["lostluck (via GitHub)" <gi...@apache.org>]

lostluck opened a new issue, #32003:
URL: https://github.com/apache/beam/issues/32003

   Java FlattenTests are failing by no longer making progress.  Largely related to "empty flattens"for the tests testEmptyFlattenAsSideInput, testFlattenPCollectionsEmpty and testFlattenPCollectionsEmptyThenParDo.
   
   Iterate by setting as the pipeline test, then running the Gradle command, pointed at a prism instance.
   
   TEST=org.apache.beam.sdk.transforms.FlattenTest
   ./gradlew :runners:portability:java:ulrLoopbackValidatesRunnerTests -PjobEndpoint=localhost:8073 --tests="$TEST"
   
   There should be at least 3 failing tests. 
   
   This seems like it should be a simple enough fix.
   
   For testEmptyFlattenAsSideInput it's probably that there's no signal that the side input is ready, preventing progress. Both the other tests testFlattenPCollectionsEmptyThenParDo and testFlattenPCollectionsEmpty uses a PAssert to check emptiness, which is still the side input issue since that's how an empty side input is detected. So it's all the same root cause.
   
   Investigation should be a replication of those test cases as equivalent Go pipelines, if they don't already exist, to eliminate potential Java or Go specific behaviors. From there, investigation should focus on whether there needs to be a "back signal" to learn if an empty pcollection is "ready" or not from the consumer side (such as for side inputs).
   
    
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [prism] Support Empty Flattens as Side Input - no way to make progress with pending elements [beam]

["damondouglas (via GitHub)" <gi...@apache.org>]

damondouglas closed issue #32003: [prism] Support Empty Flattens as Side Input - no way to make progress with pending elements
URL: https://github.com/apache/beam/issues/32003


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [prism] Support Empty Flattens as Side Input - no way to make progress with pending elements [beam]

["lostluck (via GitHub)" <gi...@apache.org>]

lostluck commented on issue #32003:
URL: https://github.com/apache/beam/issues/32003#issuecomment-2259015428

   The root issue appears to be that the interstitial flatten transform prevents a signal propagation to the side input to indicate it's ready.
   
   So the question is: how do we ensure this signal gets propagated through the element manager, which largely does this via element propagation generally?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

Re: [I] [prism] Support Empty Flattens as Side Input - no way to make progress with pending elements [beam]

["lostluck (via GitHub)" <gi...@apache.org>]

lostluck commented on issue #32003:
URL: https://github.com/apache/beam/issues/32003#issuecomment-2259306500

   Oh no. It's weirder than that.
   
   The flatten stage *has* no inputs (hence being a truly empty flatten), so nothing is ever going to trigger the watermark refresh. We can't synthetically add an impulse (the data nonce needs to go somewhere), but we can ensure that a stage without a parallel input, is automatically watermark refreshed. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org