[jira] [Created] (SPARK-9780) In case of invalid initialization of KafkaDirectStream, NPE is thrown

["Grigory Turunov (JIRA)" <ji...@apache.org>]

Grigory Turunov created SPARK-9780:
--------------------------------------

             Summary: In case of invalid initialization of KafkaDirectStream, NPE is thrown
                 Key: SPARK-9780
                 URL: https://issues.apache.org/jira/browse/SPARK-9780
             Project: Spark
          Issue Type: Bug
          Components: Streaming
    Affects Versions: 1.4.1, 1.3.1
            Reporter: Grigory Turunov
            Priority: Minor


[o.a.s.streaming.kafka.KafkaRDD.scala#L143|https://github.com/apache/spark/blob/master/external/kafka/src/main/scala/org/apache/spark/streaming/kafka/KafkaRDD.scala#L143]
In initialization of KafkaRDDIterator, there is an addition of TaskCompletionListener to the context, which calls close() to the consumer, which is not initialized yet (and will be initialized 12 lines after that).
If something happens in this 12 lines (in my case there was a private constructor for valueDecoder), an Exception, which is thrown, triggers context.markTaskCompleted() in
[o.a.s.scheduler.Task.scala#L90|https://github.com/apache/spark/blob/master/core/src/main/scala/org/apache/spark/scheduler/Task.scala#L90]
which throws NullPointerException, when tries to call close() for non-initialized consumer.
This masks original exception - so it is very hard to understand, what is happening.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org