[jira] [Created] (BEAM-6902) Beam model contract for finalization of CheckpointMark's

["Mark Norkin (JIRA)" <ji...@apache.org>]

Mark Norkin created BEAM-6902:
---------------------------------

             Summary: Beam model contract for finalization of CheckpointMark's    
                 Key: BEAM-6902
                 URL: https://issues.apache.org/jira/browse/BEAM-6902
             Project: Beam
          Issue Type: Improvement
          Components: beam-model, io-java-kafka, runner-core, runner-dataflow, sdk-java-core
            Reporter: Mark Norkin


Question: What is the contract in Beam model for when checkpoint marks shall be finalized, is there any ? 

I'm working on pipeline that reads messages from Kafka using KafkaIO, and I'm looking at _commitOffsetsInFinalize()_ option, and KafkaCheckpointMark class.

I want to achieve at-least-once message delivery semantics and want to be sure that offsets committed to Kafka after they are written to some sink.

Looking at interface of [CheckpointMark|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/UnboundedSource.CheckpointMark.html]  it's not clear when finalization shall be expected to happen.

Is it runner dependent, what to expect when executing on _DataflowRunner_ ?

And reading KafkaIO.Read javadoc on _[commitOffsetsInFinalize|[https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize--]]_ also doesn't bring clarity to my understanding, particularly the phrase 
{quote}But it does not provide *_hard processing guarantees_*
{quote}
What exactly are hard processing guarantees ?

Can I ask, please for documentation improvement in respect of _CheckpointMark_ and _commitOffsetsInFinalize_. 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)