Re: Continuous warning while consuming using new kafka-spark010 API

[Cody Koeninger <co...@koeninger.org>]

-dev +user

Than warning pretty much means what it says - the consumer tried to
get records for the given partition / offset, and couldn't do so after
polling the kafka broker for X amount of time.

If that only happens when you put additional load on Kafka via
producing, the first thing I'd do is look at what's going on on your
kafka brokers.

On Mon, Sep 19, 2016 at 11:57 AM, Nitin Goyal <ni...@gmail.com> wrote:
> Hi All,
>
> I am using the new kafka-spark010 API to consume messages from Kafka
> (brokers running kafka 0.10.0.1).
>
> I am seeing continuous following warning only when producer is writing
> messages to kafka in parallel (increased
> spark.streaming.kafka.consumer.poll.ms to 1024 ms as well) :-
>
> 16/09/19 16:44:53 WARN TaskSetManager: Lost task 97.0 in stage 32.0 (TID
> 4942, host-3): java.lang.AssertionError: assertion failed: Failed to get
> records for spark-executor-example topic2 8 1052989 after polling for 1024
>
> while at same time, I see this in spark UI corresponding to that job
> topic: topic2    partition: 8    offsets: 1051731 to 1066124
>
> Code :-
>
> val stream = KafkaUtils.createDirectStream[String, String]( ssc,
> PreferConsistent, Subscribe[String, String](topics, kafkaParams.asScala) )
>
> stream.foreachRDD {rdd => rdd.filter(_ => false).collect}
>
>
> Has anyone encountered this with the new API? Is this the expected behaviour
> or am I missing something here?
>
> --
> Regards
> Nitin Goyal

---------------------------------------------------------------------
To unsubscribe e-mail: user-unsubscribe@spark.apache.org