You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/03 23:36:39 UTC

[GitHub] [beam] kennknowles opened a new issue, #19286: FileIO errors on unbounded input with nondefault trigger

kennknowles opened a new issue, #19286:
URL: https://github.com/apache/beam/issues/19286

   `In a pipeline with unbounded input, if a user defines a custom trigger and does not specify a specific non-zero withNumShards, they may see an IllegalArgumentException at runtime due to incompatible windows.`
   
     For example, consider this compound trigger:
   
   `Window.into(new GlobalWindows())`
    `  .triggering(Repeatedly.forever(AfterFirst.of(`
    `    AfterPane.elementCountAtLeast(10000),`
    `    AfterProcessingTime.pastFirstElementInPane()`
    `              .plusDelayOf(Duration.standardMinutes(10)))))`
   
   `  .discardingFiredPanes()`
   
    
    Using that windowing without specifying sharding yields:
     
    `Inputs to Flatten had incompatible triggers:``Repeatedly.forever(AfterFirst.of(AfterPane.elementCountAtLeast(10000), AfterProcessingTime.pastFirstElementInPane().plusDelayOf(1 minute))),``Repeatedly.forever(AfterFirst.of(AfterPane.elementCountAtLeast(1), AfterSynchronizedProcessingTime.pastFirstElementInPane()))`
     
    Without explicit sharding, WriteFiles creates both a sharded and unsharded collection; the first goes through one GroupByKey while the other goes through 2. These two collections are then flattened together and they have incompatible triggers due to the double-grouped collection using a continuation trigger.
     
    If the user instead specifies numShards, then a different code path is followed that avoids this incompatibility.
     
    It looks like WriteFiles may need to be implemented differently to avoid combining collections with potentially incompatible triggers.
   
   Imported from Jira [BEAM-6399](https://issues.apache.org/jira/browse/BEAM-6399). Original Jira may contain additional context.
   Reported by: jeff.klukas@gmail.com.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [beam] cerisier commented on issue #19286: FileIO errors on unbounded input with nondefault trigger

Posted by GitBox <gi...@apache.org>.
cerisier commented on issue #19286:
URL: https://github.com/apache/beam/issues/19286#issuecomment-1328311048

   Thanks for filing the issue.
   
   It have the issue on bounded inputs as well not just unbounded (using the same windowing as in the example).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org