You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Dipesh Chandra Maurya (Jira)" <ji...@apache.org> on 2026/01/29 07:27:00 UTC

[jira] [Created] (SPARK-55271) NullPointerException in Kafka Micro-Batch Streaming Progress Reporting

Dipesh Chandra Maurya created SPARK-55271:
---------------------------------------------

             Summary: NullPointerException in Kafka Micro-Batch Streaming Progress Reporting
                 Key: SPARK-55271
                 URL: https://issues.apache.org/jira/browse/SPARK-55271
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 4.1.1
            Reporter: Dipesh Chandra Maurya


h3. Root Cause

The error originates in {{KafkaMicroBatchStream.metrics()}} at line 520, where the code attempts to call {{.get()}} on a Scala {{Option}} that contains {{{}null{}}}, then immediately invokes {{.map()}} on the null result. This violates Scala's Option contract and causes the NullPointerException.
h3. Error Flow
 # The streaming query completes a micro-batch trigger
 # Progress reporting attempts to extract source metrics via {{ProgressContext.extractSourceProgress()}}
 # The KafkaMicroBatchStream is queried for its metrics
 # An unsafe {{.get()}} call on an empty or null Option returns null
 # Attempting to call {{.map()}} on the null value throws NullPointerException

h3. Impact
 * The streaming query crashes and stops processing
 * Progress information cannot be generated for the current batch
 * The failure occurs during the finishTrigger phase, after batch processing completes

h3. Technical Details
 * {*}Component{*}: Spark Structured Streaming with Kafka source
 * {*}Class{*}: {{org.apache.spark.sql.kafka010.KafkaMicroBatchStream}}
 * {*}Method{*}: {{metrics()}} at line 520
 * {*}Trigger Type{*}: ProcessingTime (micro-batch mode)

h3. Likely Causes
 * Missing or unavailable Kafka metrics configuration
 * Empty/uninitialized metrics collection in the Kafka source
 * Race condition where metrics are accessed before initialization
 * Incompatibility between Spark and Kafka connector versions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org