[jira] [Created] (SAMZA-592) getSystemStreamMetadata loops forever when it receives bad metadata

["Chris Riccomini (JIRA)" <ji...@apache.org>]

Chris Riccomini created SAMZA-592:
-------------------------------------

             Summary: getSystemStreamMetadata loops forever when it receives bad metadata
                 Key: SAMZA-592
                 URL: https://issues.apache.org/jira/browse/SAMZA-592
             Project: Samza
          Issue Type: Bug
          Components: kafka
    Affects Versions: 0.9.0
            Reporter: Chris Riccomini
             Fix For: 0.9.0


While investigating SAMZA-576, [~ewencp] discovered a bug in the KafkaSystemAdmin that causes getSystemStreamMetadata to go into an infinite loop when it receives bad metadata from a broker. See [this|https://issues.apache.org/jira/browse/SAMZA-576?focusedCommentId=14356349&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14356349] comment.

We experienced this bug last week. We were running a healthy cluster down with topics that have a replication factor of 2. We brought down a *single* broker, and jobs would not start while the broker was down. The containers just repeated this error message:

{noformat}
  2015-02-24 22:36:43 KafkaSystemAdmin [WARN] Unable to fetch last offsets for streams [some-topic] due to kafka.common.ReplicaNotAvailableException. Retrying.
{noformat}

Checking the cluster showed that all partitions were still available, and bringing down the single broker resulted in proper leadership failover. Samza, however, was not able to start.

I was told by [~clarkhaskins] that it was actually safe to ignore the ReplicaNotAvailableException when fetching metadata. [~ewencp], can you confirm this?

It seems that there are two issues:

# KafkaSystemAdmin.getSystemStreamMetadata never refreshes data when its metadata fetch results in an error code.
# We should allow the metadata fetch to proceed, rather than throwing an exception, if there is a ReplicaNotAvailableException during metadata refreshes.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)