[jira] [Updated] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

["Amit Sela (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Amit Sela updated BEAM-744:
---------------------------
    Assignee:     (was: James Malone)

> A runner should be able to override KafkaIO max wait properties.
> ----------------------------------------------------------------
>
>                 Key: BEAM-744
>                 URL: https://issues.apache.org/jira/browse/BEAM-744
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Amit Sela
>
> KafkaIO has three "wait" properties:
> {{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the DirectRunner, and I can add that they are now limiting the Spark runner (which reads in defined time frames, which may be smaller then the wait time and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)