[jira] [Commented] (BEAM-3526) Support for checkpointMark finalize in KafkaIO

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16338206#comment-16338206 ] 

ASF GitHub Bot commented on BEAM-3526:
--------------------------------------

rangadi opened a new pull request #4481: [BEAM-3526] KakfaIO support for finalizeCheckpoint()
URL: https://github.com/apache/beam/pull/4481
 
 
   Adds an option in KafkaIO source to commit offsets to Kafka when
   `CheckpointMark#finalizecheckPoint()` is invoked. 
   
   Kafka consumer is single threaded and as a result actuall commit to Kafka might take upto 1 second (KAFKA_POLL_TIMEOUT). This is not necessarily bad since it could avoid excessive commits when multiple checkpoint marks are finalized at once by the runner.
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Support for checkpointMark finalize in KafkaIO
> ----------------------------------------------
>
>                 Key: BEAM-3526
>                 URL: https://issues.apache.org/jira/browse/BEAM-3526
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>    Affects Versions: 2.2.0
>            Reporter: Raghu Angadi
>            Assignee: Raghu Angadi
>            Priority: Major
>
> Users have been asking for option to commit offsets back to Kafka inside `CheckpointMark.finalizeCheckpoint()'. This option is some what better than setting 'AUTO_COMMIT' in Kafka consumer. 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)