[jira] [Updated] (CRUNCH-630) KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset

["Micah Whitacre (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/CRUNCH-630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Micah Whitacre updated CRUNCH-630:
----------------------------------
    Attachment: CRUNCH-630.patch

[~noslowerdna] this fit what you were thinking?

> KafkaRecordReader keeps retrying to poll data when the offset is reset to latest offset
> ---------------------------------------------------------------------------------------
>
>                 Key: CRUNCH-630
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-630
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Pooja Dhondge
>            Assignee: Micah Whitacre
>         Attachments: CRUNCH-630.patch
>
>
> We recently saw this behavior where, if the offset it is trying to read from doesn't exist on Kafka due to retention policy, the offset gets reset to latest(default) and the KafkaRecordReader keeps retrying beyond KAFKA_EMPTY_RETRY_ATTEMPTS_KEY
> {noformat}
> ...crunch.kafka.inputformat.KafkaRecordReader: No records retrieved but pending offsets to consume therefore polling again. Attempt 17/10
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)