[jira] [Created] (SAMZA-607) BrokerProxy gets stuck on down brokers

["Gian Merlino (JIRA)" <ji...@apache.org>]

Gian Merlino created SAMZA-607:
----------------------------------

             Summary: BrokerProxy gets stuck on down brokers
                 Key: SAMZA-607
                 URL: https://issues.apache.org/jira/browse/SAMZA-607
             Project: Samza
          Issue Type: Bug
    Affects Versions: 0.8.0
            Reporter: Gian Merlino


I took a broker offline for a few hours today and found that a Samza job was stuck trying to read from it while it was down, instead of switching to another broker in the ISR (this was a replicated topic with some partitions under-replicated, but all partitions available). During this time the BrokerProxy thread was in a retry loop logging a lot of ClosedChannelExceptions.

The broker had done a clean shutdown, but I think what happened is that the BrokerProxy just hadn't made any calls between when that broker stopped being leader for its partitions and when that broker went offline. So, it never got a NotLeaderForPartitionException and never abdicated.

Would it make sense for the BrokerProxy to abdicate all of its topic-partitions after getting too many network errors, and possibly shut itself down if it becomes empty? I think it'd be good to support brokers going offline temporarily or even permanently.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)