You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by "Francois Visconte (JIRA)" <ji...@apache.org> on 2015/10/16 18:15:05 UTC
[jira] [Commented] (STORM-380) Kafka spout: throw RuntimeException
if a leader cannot be found for a partition
[ https://issues.apache.org/jira/browse/STORM-380?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14960955#comment-14960955 ]
Francois Visconte commented on STORM-380:
-----------------------------------------
Hi,
We have a SLA that makes offline partition possible from time to time. This cause our topology to fail repeateadly in this case preventing processing of all other partitions.
What about adding a configuration parameter to allow a mode where offline partition are logged as warning and re-integrated in the next ZkCoordinator.refresh().
Here is the commit I made to fix our problem but I can add a config param for that.
Cheers,
F.
> Kafka spout: throw RuntimeException if a leader cannot be found for a partition
> -------------------------------------------------------------------------------
>
> Key: STORM-380
> URL: https://issues.apache.org/jira/browse/STORM-380
> Project: Apache Storm
> Issue Type: Improvement
> Components: storm-kafka
> Reporter: Stephen Elliott
> Assignee: Stephen Elliott
> Priority: Minor
> Fix For: 0.9.3
>
>
> If one of the partitions of the Kafka queue has no leader available, DynamicBrokersReader.getLeaderFor(partition) returns -1. This is logged in DynamicBrokersReader.getBrokerInfo (as there's no ZK node for a broker id -1) but there is no further action.
> In this situation the spout will carry on emitting only for those partitions that are available.
> My assumption is that if some partitions for a queue are unavailable, there's a problem that needs to be addressed before processing continues. Happy to discuss further.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)