[jira] [Commented] (BEAM-744) A runner should be able to override KafkaIO max wait properties.

["Raghu Angadi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15569856#comment-15569856 ] 

Raghu Angadi commented on BEAM-744:
-----------------------------------

> KAFKA_POLL_TIMEOUT - consumer poll timeout, default: 1 second.
This timeout is KafkaIO internal implementation detail and should be ignored here. It does not impose any limitations on the reader (i.e. the reader can be closed before this timeout and everything is cleaned up properly).

> A runner should be able to override KafkaIO max wait properties.
> ----------------------------------------------------------------
>
>                 Key: BEAM-744
>                 URL: https://issues.apache.org/jira/browse/BEAM-744
>             Project: Beam
>          Issue Type: Bug
>          Components: sdk-java-extensions
>            Reporter: Amit Sela
>
> KafkaIO has three "wait" properties:
> {{KAFKA_POLL_TIMEOUT}} - consumer poll timeout, default: 1 second.
> {{START_NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, default: 5 seconds.
> {{NEW_RECORDS_POLL_TIMEOUT}} - wait for new records inside {{start()}}, default: 10 msec.
> [~rangadi] mentioned some of these were set to due to limitations of the DirectRunner, and I can add that they are now limiting the Spark runner (which reads in defined time frames, which may be smaller then the wait time and so never actually read).
> This feels like defaults should be set for optimal read from Kafka, while a runner may override those if it needs to.
> [~rangadi] also mentioned that this could be set in {{PipelineOptions}} which may be passed when creating the reader. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)