You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by unk1102 <um...@gmail.com> on 2015/07/29 19:38:55 UTC
Spark Streaming Kafka could not find leader offset for Set()
Hi I have Spark Streaming code which streams from Kafka topic it used to work
fine but suddenly it started throwing the following exception
Exception in thread "main" org.apache.spark.SparkException:
org.apache.spark.SparkException: Couldn't find leader offsets for Set()
at
org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
at
org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
at scala.util.Either.fold(Either.scala:97)
at
org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
at
org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
at
org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
My Spark Streaming client code is very simple I just create one receiver
using the following code and trying to print messages it consumed
JavaPairInputDStream<String, String> messages =
KafkaUtils.createDirectStream(jssc,
String.class,
String.class,
StringDecoder.class,
StringDecoder.class,
kafkaParams,
topicSet);
Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic has
data I can see data using other Kafka consumers but above Spark Streaming
code throws exception saying leader offset not found. I tried both smallest
and largest offset. I wonder what happened this code used to work earlier. I
am using Spark-Streaming 1.3.1 as it was working in this version I tried in
1.4.1 and same exception. Please guide. I am new to Spark thanks in advance.
--
View this message in context: http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
Sent from the Apache Spark User List mailing list archive at Nabble.com.
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by gaurav sharma <sh...@gmail.com>.
I have run into similar excpetions
ERROR DirectKafkaInputDStream: ArrayBuffer(java.net.SocketTimeoutException,
org.apache.spark.SparkException: Couldn't find leader offsets for
Set([AdServe,1]))
and the issue has happened on Kafka Side, where my broker offsets go out of
sync, or do not return leader for this particular partition
./bin/kafka-run-class.sh kafka.tools.GetOffsetShell --topic AdServe
--broker-list BROKER_IP --time -1
this shall return u valid offsets for all your kafka partitons
On Thu, Jul 30, 2015 at 7:58 PM, Umesh Kacha <um...@gmail.com> wrote:
> Hi Cody sorry my bad you were right there was a typo in topicSet. When I
> corrected typo in topicSet it started working. Thanks a lot.
>
> Regards
>
> On Thu, Jul 30, 2015 at 7:43 PM, Cody Koeninger <co...@koeninger.org>
> wrote:
>
>> Can you post the code including the values of kafkaParams and topicSet,
>> ideally the relevant output of kafka-topics.sh --describe as well
>>
>> On Wed, Jul 29, 2015 at 11:39 PM, Umesh Kacha <um...@gmail.com>
>> wrote:
>>
>>> Hi thanks for the response. Like I already mentioned in the question
>>> kafka topic is valid and it has data I can see data in it using another
>>> kafka consumer.
>>> On Jul 30, 2015 7:31 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
>>>
>>>> The last time someone brought this up on the mailing list, the issue
>>>> actually was that the topic(s) didn't exist in Kafka at the time the spark
>>>> job was running.
>>>>
>>>>
>>>>
>>>>
>>>>
>>>> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <td...@databricks.com>
>>>> wrote:
>>>>
>>>>> There is a known issue that Kafka cannot return leader if there is not
>>>>> data in the topic. I think it was raised in another thread in this forum.
>>>>> Is that the issue?
>>>>>
>>>>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi I have Spark Streaming code which streams from Kafka topic it used
>>>>>> to work
>>>>>> fine but suddenly it started throwing the following exception
>>>>>>
>>>>>> Exception in thread "main" org.apache.spark.SparkException:
>>>>>> org.apache.spark.SparkException: Couldn't find leader offsets for
>>>>>> Set()
>>>>>> at
>>>>>>
>>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>>>> at
>>>>>>
>>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>>>> at scala.util.Either.fold(Either.scala:97)
>>>>>> at
>>>>>>
>>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
>>>>>> at
>>>>>>
>>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
>>>>>> at
>>>>>>
>>>>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>>>>>> My Spark Streaming client code is very simple I just create one
>>>>>> receiver
>>>>>> using the following code and trying to print messages it consumed
>>>>>>
>>>>>> JavaPairInputDStream<String, String> messages =
>>>>>> KafkaUtils.createDirectStream(jssc,
>>>>>> String.class,
>>>>>> String.class,
>>>>>> StringDecoder.class,
>>>>>> StringDecoder.class,
>>>>>> kafkaParams,
>>>>>> topicSet);
>>>>>>
>>>>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka
>>>>>> topic has
>>>>>> data I can see data using other Kafka consumers but above Spark
>>>>>> Streaming
>>>>>> code throws exception saying leader offset not found. I tried both
>>>>>> smallest
>>>>>> and largest offset. I wonder what happened this code used to work
>>>>>> earlier. I
>>>>>> am using Spark-Streaming 1.3.1 as it was working in this version I
>>>>>> tried in
>>>>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
>>>>>> advance.
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> View this message in context:
>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>> Nabble.com.
>>>>>>
>>>>>> ---------------------------------------------------------------------
>>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>>
>>>>>>
>>>>>
>>>>
>>
>
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by Umesh Kacha <um...@gmail.com>.
Hi Cody sorry my bad you were right there was a typo in topicSet. When I
corrected typo in topicSet it started working. Thanks a lot.
Regards
On Thu, Jul 30, 2015 at 7:43 PM, Cody Koeninger <co...@koeninger.org> wrote:
> Can you post the code including the values of kafkaParams and topicSet,
> ideally the relevant output of kafka-topics.sh --describe as well
>
> On Wed, Jul 29, 2015 at 11:39 PM, Umesh Kacha <um...@gmail.com>
> wrote:
>
>> Hi thanks for the response. Like I already mentioned in the question
>> kafka topic is valid and it has data I can see data in it using another
>> kafka consumer.
>> On Jul 30, 2015 7:31 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
>>
>>> The last time someone brought this up on the mailing list, the issue
>>> actually was that the topic(s) didn't exist in Kafka at the time the spark
>>> job was running.
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <td...@databricks.com>
>>> wrote:
>>>
>>>> There is a known issue that Kafka cannot return leader if there is not
>>>> data in the topic. I think it was raised in another thread in this forum.
>>>> Is that the issue?
>>>>
>>>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi I have Spark Streaming code which streams from Kafka topic it used
>>>>> to work
>>>>> fine but suddenly it started throwing the following exception
>>>>>
>>>>> Exception in thread "main" org.apache.spark.SparkException:
>>>>> org.apache.spark.SparkException: Couldn't find leader offsets for Set()
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>>> at scala.util.Either.fold(Either.scala:97)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
>>>>> at
>>>>>
>>>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>>>>> My Spark Streaming client code is very simple I just create one
>>>>> receiver
>>>>> using the following code and trying to print messages it consumed
>>>>>
>>>>> JavaPairInputDStream<String, String> messages =
>>>>> KafkaUtils.createDirectStream(jssc,
>>>>> String.class,
>>>>> String.class,
>>>>> StringDecoder.class,
>>>>> StringDecoder.class,
>>>>> kafkaParams,
>>>>> topicSet);
>>>>>
>>>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka
>>>>> topic has
>>>>> data I can see data using other Kafka consumers but above Spark
>>>>> Streaming
>>>>> code throws exception saying leader offset not found. I tried both
>>>>> smallest
>>>>> and largest offset. I wonder what happened this code used to work
>>>>> earlier. I
>>>>> am using Spark-Streaming 1.3.1 as it was working in this version I
>>>>> tried in
>>>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
>>>>> advance.
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> View this message in context:
>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>> Nabble.com.
>>>>>
>>>>> ---------------------------------------------------------------------
>>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>>
>>>>>
>>>>
>>>
>
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by Cody Koeninger <co...@koeninger.org>.
Can you post the code including the values of kafkaParams and topicSet,
ideally the relevant output of kafka-topics.sh --describe as well
On Wed, Jul 29, 2015 at 11:39 PM, Umesh Kacha <um...@gmail.com> wrote:
> Hi thanks for the response. Like I already mentioned in the question kafka
> topic is valid and it has data I can see data in it using another kafka
> consumer.
> On Jul 30, 2015 7:31 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
>
>> The last time someone brought this up on the mailing list, the issue
>> actually was that the topic(s) didn't exist in Kafka at the time the spark
>> job was running.
>>
>>
>>
>>
>>
>> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <td...@databricks.com>
>> wrote:
>>
>>> There is a known issue that Kafka cannot return leader if there is not
>>> data in the topic. I think it was raised in another thread in this forum.
>>> Is that the issue?
>>>
>>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com> wrote:
>>>
>>>> Hi I have Spark Streaming code which streams from Kafka topic it used
>>>> to work
>>>> fine but suddenly it started throwing the following exception
>>>>
>>>> Exception in thread "main" org.apache.spark.SparkException:
>>>> org.apache.spark.SparkException: Couldn't find leader offsets for Set()
>>>> at
>>>>
>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>> at
>>>>
>>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>>> at scala.util.Either.fold(Either.scala:97)
>>>> at
>>>>
>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
>>>> at
>>>>
>>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
>>>> at
>>>>
>>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>>>> My Spark Streaming client code is very simple I just create one receiver
>>>> using the following code and trying to print messages it consumed
>>>>
>>>> JavaPairInputDStream<String, String> messages =
>>>> KafkaUtils.createDirectStream(jssc,
>>>> String.class,
>>>> String.class,
>>>> StringDecoder.class,
>>>> StringDecoder.class,
>>>> kafkaParams,
>>>> topicSet);
>>>>
>>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka
>>>> topic has
>>>> data I can see data using other Kafka consumers but above Spark
>>>> Streaming
>>>> code throws exception saying leader offset not found. I tried both
>>>> smallest
>>>> and largest offset. I wonder what happened this code used to work
>>>> earlier. I
>>>> am using Spark-Streaming 1.3.1 as it was working in this version I
>>>> tried in
>>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
>>>> advance.
>>>>
>>>>
>>>>
>>>>
>>>> --
>>>> View this message in context:
>>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
>>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>>> For additional commands, e-mail: user-help@spark.apache.org
>>>>
>>>>
>>>
>>
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by Umesh Kacha <um...@gmail.com>.
Hi thanks for the response. Like I already mentioned in the question kafka
topic is valid and it has data I can see data in it using another kafka
consumer.
On Jul 30, 2015 7:31 AM, "Cody Koeninger" <co...@koeninger.org> wrote:
> The last time someone brought this up on the mailing list, the issue
> actually was that the topic(s) didn't exist in Kafka at the time the spark
> job was running.
>
>
>
>
>
> On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <td...@databricks.com>
> wrote:
>
>> There is a known issue that Kafka cannot return leader if there is not
>> data in the topic. I think it was raised in another thread in this forum.
>> Is that the issue?
>>
>> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com> wrote:
>>
>>> Hi I have Spark Streaming code which streams from Kafka topic it used to
>>> work
>>> fine but suddenly it started throwing the following exception
>>>
>>> Exception in thread "main" org.apache.spark.SparkException:
>>> org.apache.spark.SparkException: Couldn't find leader offsets for Set()
>>> at
>>>
>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>> at
>>>
>>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>>> at scala.util.Either.fold(Either.scala:97)
>>> at
>>>
>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
>>> at
>>>
>>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
>>> at
>>>
>>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>>> My Spark Streaming client code is very simple I just create one receiver
>>> using the following code and trying to print messages it consumed
>>>
>>> JavaPairInputDStream<String, String> messages =
>>> KafkaUtils.createDirectStream(jssc,
>>> String.class,
>>> String.class,
>>> StringDecoder.class,
>>> StringDecoder.class,
>>> kafkaParams,
>>> topicSet);
>>>
>>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic
>>> has
>>> data I can see data using other Kafka consumers but above Spark Streaming
>>> code throws exception saying leader offset not found. I tried both
>>> smallest
>>> and largest offset. I wonder what happened this code used to work
>>> earlier. I
>>> am using Spark-Streaming 1.3.1 as it was working in this version I tried
>>> in
>>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
>>> advance.
>>>
>>>
>>>
>>>
>>> --
>>> View this message in context:
>>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
>>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by Cody Koeninger <co...@koeninger.org>.
The last time someone brought this up on the mailing list, the issue
actually was that the topic(s) didn't exist in Kafka at the time the spark
job was running.
On Wed, Jul 29, 2015 at 6:17 PM, Tathagata Das <td...@databricks.com> wrote:
> There is a known issue that Kafka cannot return leader if there is not
> data in the topic. I think it was raised in another thread in this forum.
> Is that the issue?
>
> On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com> wrote:
>
>> Hi I have Spark Streaming code which streams from Kafka topic it used to
>> work
>> fine but suddenly it started throwing the following exception
>>
>> Exception in thread "main" org.apache.spark.SparkException:
>> org.apache.spark.SparkException: Couldn't find leader offsets for Set()
>> at
>>
>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>> at
>>
>> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
>> at scala.util.Either.fold(Either.scala:97)
>> at
>>
>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
>> at
>>
>> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
>> at
>>
>> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
>> My Spark Streaming client code is very simple I just create one receiver
>> using the following code and trying to print messages it consumed
>>
>> JavaPairInputDStream<String, String> messages =
>> KafkaUtils.createDirectStream(jssc,
>> String.class,
>> String.class,
>> StringDecoder.class,
>> StringDecoder.class,
>> kafkaParams,
>> topicSet);
>>
>> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic
>> has
>> data I can see data using other Kafka consumers but above Spark Streaming
>> code throws exception saying leader offset not found. I tried both
>> smallest
>> and largest offset. I wonder what happened this code used to work
>> earlier. I
>> am using Spark-Streaming 1.3.1 as it was working in this version I tried
>> in
>> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
>> advance.
>>
>>
>>
>>
>> --
>> View this message in context:
>> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
>> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>> For additional commands, e-mail: user-help@spark.apache.org
>>
>>
>
Re: Spark Streaming Kafka could not find leader offset for Set()
Posted by Tathagata Das <td...@databricks.com>.
There is a known issue that Kafka cannot return leader if there is not data
in the topic. I think it was raised in another thread in this forum. Is
that the issue?
On Wed, Jul 29, 2015 at 10:38 AM, unk1102 <um...@gmail.com> wrote:
> Hi I have Spark Streaming code which streams from Kafka topic it used to
> work
> fine but suddenly it started throwing the following exception
>
> Exception in thread "main" org.apache.spark.SparkException:
> org.apache.spark.SparkException: Couldn't find leader offsets for Set()
> at
>
> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
> at
>
> org.apache.spark.streaming.kafka.KafkaUtils$$anonfun$createDirectStream$2.apply(KafkaUtils.scala:413)
> at scala.util.Either.fold(Either.scala:97)
> at
>
> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:412)
> at
>
> org.apache.spark.streaming.kafka.KafkaUtils$.createDirectStream(KafkaUtils.scala:528)
> at
>
> org.apache.spark.streaming.kafka.KafkaUtils.createDirectStream(KafkaUtils.scala)
> My Spark Streaming client code is very simple I just create one receiver
> using the following code and trying to print messages it consumed
>
> JavaPairInputDStream<String, String> messages =
> KafkaUtils.createDirectStream(jssc,
> String.class,
> String.class,
> StringDecoder.class,
> StringDecoder.class,
> kafkaParams,
> topicSet);
>
> Kafka param is only one I specify kafka.ofset.reset=largest. Kafka topic
> has
> data I can see data using other Kafka consumers but above Spark Streaming
> code throws exception saying leader offset not found. I tried both smallest
> and largest offset. I wonder what happened this code used to work earlier.
> I
> am using Spark-Streaming 1.3.1 as it was working in this version I tried in
> 1.4.1 and same exception. Please guide. I am new to Spark thanks in
> advance.
>
>
>
>
> --
> View this message in context:
> http://apache-spark-user-list.1001560.n3.nabble.com/Spark-Streaming-Kafka-could-not-find-leader-offset-for-Set-tp24066.html
> Sent from the Apache Spark User List mailing list archive at Nabble.com.
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>