[GitHub] [beam] kkdoon commented on issue #28554: [Bug]: Unable to drain Flink job when RequiresStableInput is used

["kkdoon (via GitHub)" <gi...@apache.org>]

kkdoon commented on issue #28554:
URL: https://github.com/apache/beam/issues/28554#issuecomment-1730374841

   I have tried stopping the job using both the [CLI](https://github.com/apache/flink/blob/5682472e4029c25d4a2651a609c999029fa3281b/flink-clients/src/main/java/org/apache/flink/client/cli/CliFrontend.java#L541) (with --drain) as well as via flink [RestClusterClient library ](https://nightlies.apache.org/flink/flink-docs-master/api/java/org/apache/flink/client/program/rest/RestClusterClient.html#stopWithSavepoint-org.apache.flink.api.common.JobID-boolean-java.lang.String-org.apache.flink.core.execution.SavepointFormatType-) where `advanceToEndOfTime` is set to True to drain the job.
   
   Note that when drain is set to false, pipeline is able to successfully stop with savepoint and DoFn with `RequiresStableInput` annotation does emit the buffered results. There's no error since [flushData](https://github.com/apache/beam/blob/79d0a8d11095522a036a6b3007f5fed4f6f46b3b/runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/DoFnOperator.java#L608) is not invoked & watermark does not proceed to MAX_WATERMARK.
   
   So this issue is only for the specific scenario when drain flag is used to advance the watermark and permanently stop the pipeline.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org