Re: [I] [Bug]: Unable to drain Flink job when RequiresStableInput is used [beam]

["kkdoon (via GitHub)" <gi...@apache.org>]

kkdoon commented on issue #28554:
URL: https://github.com/apache/beam/issues/28554#issuecomment-1745550628

   No worries and thanks for your response.
   
   I wanted to double check that if we flush the buffer within `flushData` AND before final checkpoint operation is complete, won't that violate stable input DoFn contract (where stable DoFn should only be invoked once state is check-pointed)?  Or is it ok in case of drain operation? I might be misunderstanding the sequence of operations here.
   
   Issue i see is that once we flush the buffer, it would trigger the downstream DoFn processing. And in the scenario where the downstream stable DoFn operation fails (say its writing to a sink and the sink is unavailable and/or checkpoint itself has timeout), our final checkpoint and drain operation will fail, which could lead to inconsistent state.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org