You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "yue long (JIRA)" <ji...@apache.org> on 2017/08/25 08:57:00 UTC
[jira] [Commented] (SPARK-18983) Couldn't find leader offsets
exception when the one of kafka cluster brokers is down
[ https://issues.apache.org/jira/browse/SPARK-18983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16141388#comment-16141388 ]
yue long commented on SPARK-18983:
----------------------------------
Hi [~srowen], sorry for openning a new ticket for kakfa broker failure.
I know it is not the bug of spark streaming.
As we know, kafka will take only a few seconds to substitute the failed broker with an alive one, but the streaming will cost at least 3~5 minutes to restart the failed streaming job. I think it is not friendly for kafka Dstream user, and I hope some improvement could be made for more stability of kafka Dstream. (That is why I open the duplicated ticket with label "improvement", not "bug")
If we apply some retry logic or other method, this will make our streaming job robuster. This small improvement will help a lot of spark users.
Thank you
> Couldn't find leader offsets exception when the one of kafka cluster brokers is down
> ------------------------------------------------------------------------------------
>
> Key: SPARK-18983
> URL: https://issues.apache.org/jira/browse/SPARK-18983
> Project: Spark
> Issue Type: Bug
> Reporter: kraken
> Priority: Critical
>
> Hello,i got a trouble today
> when the company's PE restarts the one of kafka cluster broker,my spark job run failed.
> Exception in thread "main" org.apache.spark.SparkException: get earliest leader offsets failed: ArrayBuffer(org.apache.spark.SparkException: Couldn't find leaders for Set
> I use the low level type to consume kafka log, i know that high level can be aware of the lead changes, but how can i do that by use low level?
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org