[jira] [Commented] (SAMZA-584) KafkaSystemConsumer race condition

["Chris Riccomini (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SAMZA-584?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14345251#comment-14345251 ] 

Chris Riccomini commented on SAMZA-584:
---------------------------------------

[~closeuris], could I get a +1 when you have some time?

> KafkaSystemConsumer race condition
> ----------------------------------
>
>                 Key: SAMZA-584
>                 URL: https://issues.apache.org/jira/browse/SAMZA-584
>             Project: Samza
>          Issue Type: Bug
>          Components: kafka
>    Affects Versions: 0.9.0
>            Reporter: Chris Riccomini
>            Assignee: Chris Riccomini
>             Fix For: 0.9.0
>
>         Attachments: SAMZA-584-0.patch
>
>
> After SAMZA-579, a race condition was introduced that leads to an NPE (effectively). The error is that this log line repeats over and over:
> {noformat}
> 2015-03-02 23:59:36 KafkaSystemConsumer [WARN] While refreshing brokers for [topic1,5]: java.util.NoSuchElementException: None.get. Retrying.
> {noformat}
> This is caused by:
> {code}
>               val nextOffset = topicPartitionsAndOffsets.get(head).get
> {code}
> Which is called *outside* this if statement:
> {code}
>             if (topicPartitionsAndOffsets.contains(head)) {
> {code}
> If multiple threads call KafkaSystemConsumer.refreshBrokers, there is a race condition where one might get a TopicAndPartition that's already been removed from topicPartitionsAndOffsets. This would cause the .get call to be None.get, which will fail.
> This error is seen immediately when consuming from a large number of brokers, as every BrokerProxy thread immediately refreshes on start. There are so many threads that one is bound to catch this error.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)