[jira] [Commented] (BEAM-6285) option to disable offsetConsumer in KafkaIO

["Raghu Angadi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-6285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16726486#comment-16726486 ] 

Raghu Angadi commented on BEAM-6285:
------------------------------------

Could you expand a bit more on the warning? Is it because of randomized group_id? May be simpler solution is to let user provide a group id for 'offset consumer'?

In addition to Backlog, watermarks are also dependent on offset consumer (e.g. logAppendTimestampPolicy advances timestamp if the reader is caught up).

> option to disable offsetConsumer in KafkaIO
> -------------------------------------------
>
>                 Key: BEAM-6285
>                 URL: https://issues.apache.org/jira/browse/BEAM-6285
>             Project: Beam
>          Issue Type: Improvement
>          Components: io-java-kafka
>            Reporter: Xu Mingmin
>            Assignee: Xu Mingmin
>            Priority: Major
>
> Add an option in KafkaIO.read() to avoid using `offsetConsumer`. 
> *What's the problem:*
> When security is enabled, function `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN log.
> *What's the cause:*
> `offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it doesn't work as expected when security is enabled. In our case, `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer is approved.
> *What's the solution:*
> Add an option to disable `offsetConsumer` by developers;
> *What's the side effect:*
> With `offsetConsumer` disabled, backlog is unknown.
>  
> [~rangadi] any comments?
>  
>  



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)