You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Xu Mingmin (JIRA)" <ji...@apache.org> on 2018/12/20 19:05:00 UTC

[jira] [Created] (BEAM-6285) option to disable offsetConsumer in KafkaIO

Xu Mingmin created BEAM-6285:
--------------------------------

             Summary: option to disable offsetConsumer in KafkaIO
                 Key: BEAM-6285
                 URL: https://issues.apache.org/jira/browse/BEAM-6285
             Project: Beam
          Issue Type: Improvement
          Components: io-java-kafka
            Reporter: Xu Mingmin
            Assignee: Xu Mingmin


Add an option in KafkaIO.read() to avoid using `offsetConsumer`. 

*What's the problem:*

When security is enabled, function `KafkaUnboundedReader.updateLatestOffsets()` always fails with lots of WARN log.

*What's the cause:*

`offsetConsumer` uses a mocked `ConsumerConfig.GROUP_ID_CONFIG`, and it doesn't work as expected when security is enabled. In our case, `ConsumerConfig.GROUP_ID_CONFIG` is the key to validate whether this consumer is approved.

*What's the solution:*

Add an option to disable `offsetConsumer` by developers;

*What's the side effect:*

With `offsetConsumer` disabled, backlog is unknown.

 

[~rangadi] any comments?

 

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)