You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Bryan Jeffrey <br...@gmail.com> on 2016/06/13 13:25:20 UTC
Kafka Exceptions
All,
We're running a Spark job that is consuming data from a large Kafka cluster
using the Direct Stream receiver. We're seeing intermittent
NotLeaderForPartitionExceptions when the leader is moved to another
broker. Currently even with retry enabled we're seeing the job fail at
this exception. Is there a configuration setting I am missing? How are
these issues typically handled?
User class threw exception: org.apache.spark.SparkException:
ArrayBuffer(kafka.common.NotLeaderForPartitionException,
org.apache.spark.SparkException: Couldn't find leader offsets for
Set([MyTopic,43]))
Thank you,
Bryan Jeffrey
Re: Kafka Exceptions
Posted by Cody Koeninger <co...@koeninger.org>.
Is the exception on the driver or the executor?
To be clear, you're going to see a task fail if a partition changes
leader while the task is running, regardless of configuration
settings. The task should be retried up the maxFailures though.
What are maxRetries and maxFailures set to? How long does the leader
change take? How many task retries happen during that time (assuming
it's an exception on the executor)?
On Mon, Jun 13, 2016 at 10:40 AM, Bryan Jeffrey <br...@gmail.com> wrote:
> Cody,
>
> We already set the maxRetries. We're still seeing issue - when leader is
> shifted, for example, it does not appear that direct stream reader correctly
> handles this. We're running 1.6.1.
>
> Bryan Jeffrey
>
> On Mon, Jun 13, 2016 at 10:37 AM, Cody Koeninger <co...@koeninger.org> wrote:
>>
>> http://spark.apache.org/docs/latest/configuration.html
>>
>> spark.streaming.kafka.maxRetries
>>
>> spark.task.maxFailures
>>
>> On Mon, Jun 13, 2016 at 8:25 AM, Bryan Jeffrey <br...@gmail.com>
>> wrote:
>> > All,
>> >
>> > We're running a Spark job that is consuming data from a large Kafka
>> > cluster
>> > using the Direct Stream receiver. We're seeing intermittent
>> > NotLeaderForPartitionExceptions when the leader is moved to another
>> > broker.
>> > Currently even with retry enabled we're seeing the job fail at this
>> > exception. Is there a configuration setting I am missing? How are
>> > these
>> > issues typically handled?
>> >
>> > User class threw exception: org.apache.spark.SparkException:
>> > ArrayBuffer(kafka.common.NotLeaderForPartitionException,
>> > org.apache.spark.SparkException: Couldn't find leader offsets for
>> > Set([MyTopic,43]))
>> >
>> > Thank you,
>> >
>> > Bryan Jeffrey
>> >
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: Kafka Exceptions
Posted by Bryan Jeffrey <br...@gmail.com>.
Cody,
We already set the maxRetries. We're still seeing issue - when leader is
shifted, for example, it does not appear that direct stream reader
correctly handles this. We're running 1.6.1.
Bryan Jeffrey
On Mon, Jun 13, 2016 at 10:37 AM, Cody Koeninger <co...@koeninger.org> wrote:
> http://spark.apache.org/docs/latest/configuration.html
>
> spark.streaming.kafka.maxRetries
>
> spark.task.maxFailures
>
> On Mon, Jun 13, 2016 at 8:25 AM, Bryan Jeffrey <br...@gmail.com>
> wrote:
> > All,
> >
> > We're running a Spark job that is consuming data from a large Kafka
> cluster
> > using the Direct Stream receiver. We're seeing intermittent
> > NotLeaderForPartitionExceptions when the leader is moved to another
> broker.
> > Currently even with retry enabled we're seeing the job fail at this
> > exception. Is there a configuration setting I am missing? How are these
> > issues typically handled?
> >
> > User class threw exception: org.apache.spark.SparkException:
> > ArrayBuffer(kafka.common.NotLeaderForPartitionException,
> > org.apache.spark.SparkException: Couldn't find leader offsets for
> > Set([MyTopic,43]))
> >
> > Thank you,
> >
> > Bryan Jeffrey
> >
>
Re: Kafka Exceptions
Posted by Cody Koeninger <co...@koeninger.org>.
http://spark.apache.org/docs/latest/configuration.html
spark.streaming.kafka.maxRetries
spark.task.maxFailures
On Mon, Jun 13, 2016 at 8:25 AM, Bryan Jeffrey <br...@gmail.com> wrote:
> All,
>
> We're running a Spark job that is consuming data from a large Kafka cluster
> using the Direct Stream receiver. We're seeing intermittent
> NotLeaderForPartitionExceptions when the leader is moved to another broker.
> Currently even with retry enabled we're seeing the job fail at this
> exception. Is there a configuration setting I am missing? How are these
> issues typically handled?
>
> User class threw exception: org.apache.spark.SparkException:
> ArrayBuffer(kafka.common.NotLeaderForPartitionException,
> org.apache.spark.SparkException: Couldn't find leader offsets for
> Set([MyTopic,43]))
>
> Thank you,
>
> Bryan Jeffrey
>
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org