[jira] [Work logged] (BEAM-3925) Allow ValueProvider for KafkaIO

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-3925?focusedWorklogId=153255&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153255 ]

ASF GitHub Bot logged work on BEAM-3925:
----------------------------------------

                Author: ASF GitHub Bot
            Created on: 10/Oct/18 19:52
            Start Date: 10/Oct/18 19:52
    Worklog Time Spent: 10m 
      Work Description: rangadi opened a new pull request #6636: [BEAM-3925] [DO NOT MERGE] KafkaIO : Value provider support for reader configuration. 
URL: https://github.com/apache/beam/pull/6636
 
 
   ValueProvider support for KafkaIO reader configuration (input topics & bootstrap servers).
   [NOTE: This is not meant for merge into master yet]
   
   This enables users to write Dataflow templates using KafkaIO.
   
   The main funcational changes is that partition information for topics is fetched at runtime by each of the source splits just before reading. This is required since Kafka topic names and cluster configuration may not be available at graph construction time.
   
   This implies that we cannot adjust number of splits based on number of Kafka partitions. In order to get around this limitation this lets users explicitly specify number of splits (`withNumSplits()`). We can handle this better if we can let some splits not to have any partitions assigned at runtime. We need to something like that to be able to merge this into master. 
   
   For now, this PR will serve as good starting point for developing templates with KafkaIO.
   [cc: @azurezyq, @pupamanyu] 

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Issue Time Tracking
-------------------

    Worklog Id:     (was: 153255)
    Time Spent: 6h 40m  (was: 6.5h)

> Allow ValueProvider for KafkaIO
> -------------------------------
>
>                 Key: BEAM-3925
>                 URL: https://issues.apache.org/jira/browse/BEAM-3925
>             Project: Beam
>          Issue Type: Improvement
>          Components: sdk-java-core
>            Reporter: Sameer Abhyankar
>            Assignee: Pramod Upamanyu
>            Priority: Major
>          Time Spent: 6h 40m
>  Remaining Estimate: 0h
>
> Add ValueProvider support for the various methods in KafkaIO. This would allow us to use KafkaIO in reusable pipeline templates.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)