You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "peay (JIRA)" <ji...@apache.org> on 2017/06/14 16:44:00 UTC

[jira] [Created] (BEAM-2449) Support ValueProviders in KafkaIO

peay created BEAM-2449:
--------------------------

             Summary: Support ValueProviders in KafkaIO
                 Key: BEAM-2449
                 URL: https://issues.apache.org/jira/browse/BEAM-2449
             Project: Beam
          Issue Type: Improvement
          Components: runner-dataflow, sdk-java-extensions
    Affects Versions: 2.0.0
            Reporter: peay
            Assignee: Thomas Groh
            Priority: Minor


KafkaIO takes parameters at construction time:
- When partitions are provided, KafkaIO actually does nothing particular at construction time.
- When only a topic is provided, I believe KafkaIO does fetch the list of partitions at construction time.

Using ValueProvider for the brokers would be a good improvement. In particular, when partitions are provided, a RuntimeProvider could be passed in.

Being able to always use a RuntimeProvider would be even better, such as when using {{TextIO.xxx.withoutValidation()}}: this would considerably improve the workflow of using Dataflow templates with KafkaIO, which currently requires to hardcode brokers into the template.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)