You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Mark Norkin (JIRA)" <ji...@apache.org> on 2019/04/16 14:53:00 UTC
[jira] [Updated] (BEAM-6902) Beam model contract for finalization
of CheckpointMark's
[ https://issues.apache.org/jira/browse/BEAM-6902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mark Norkin updated BEAM-6902:
------------------------------
Attachment: beam-kafka-io-commit-model-examples-master.zip
> Beam model contract for finalization of CheckpointMark's
> ------------------------------------------------------------
>
> Key: BEAM-6902
> URL: https://issues.apache.org/jira/browse/BEAM-6902
> Project: Beam
> Issue Type: Improvement
> Components: beam-model, io-java-kafka, runner-core, runner-dataflow, sdk-java-core
> Reporter: Mark Norkin
> Priority: Major
> Labels: documentation
> Attachments: beam-kafka-io-commit-model-examples-master.zip
>
>
> Question: What is the contract in Beam model for when checkpoint marks shall be finalized, is there any ?
> I'm working on pipeline that reads messages from Kafka using KafkaIO, and I'm looking at _commitOffsetsInFinalize()_ option, and KafkaCheckpointMark class.
> I want to achieve at-least-once message delivery semantics and want to be sure that offsets committed to Kafka after they are written to some sink.
> Looking at interface of [CheckpointMark|https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/UnboundedSource.CheckpointMark.html] it's not clear when finalization shall be expected to happen.
> Is it runner dependent, what to expect when executing on _DataflowRunner_ ?
> And reading KafkaIO.Read javadoc on _commitOffsetsInFinalize_
> _[https://beam.apache.org/releases/javadoc/2.9.0/org/apache/beam/sdk/io/kafka/KafkaIO.Read.html#commitOffsetsInFinalize--]_
> also doesn't bring clarity to my understanding, particularly the phrase
> {quote}But it does not provide *_hard processing guarantees_*
> {quote}
> What exactly are hard processing guarantees ?
> Can I ask, please for documentation improvement in respect of _CheckpointMark_ and _commitOffsetsInFinalize_.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)