[jira] [Commented] (SPARK-55271) NullPointerException in Kafka Micro-Batch Streaming Progress Reporting

["Charl P. Botha (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-55271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18062540#comment-18062540 ] 

Charl P. Botha commented on SPARK-55271:
----------------------------------------

We ran into this bug in production after upgrading our streaming pipelines from Spark 3.5.x to 4.1.x

With Claude's help, we implemented the same fix as in the linked PR, with the only difference that we changed `Some(...)` to `Option(...)` in the RTM path as well – not sure if it's required there though.

We built the patched spark-sql-kafka-0-10 jar and patched it into the relevant pipelines, which are now running without the NPE.

Would be great to see this work reviewed and merged!

> NullPointerException in Kafka Micro-Batch Streaming Progress Reporting
> ----------------------------------------------------------------------
>
>                 Key: SPARK-55271
>                 URL: https://issues.apache.org/jira/browse/SPARK-55271
>             Project: Spark
>          Issue Type: Bug
>          Components: Structured Streaming
>    Affects Versions: 4.1.1
>            Reporter: Dipesh Chandra Maurya
>            Priority: Critical
>              Labels: pull-request-available
>
> h3. Root Cause
> The error originates in {{KafkaMicroBatchStream.metrics()}} at line 520, where the code attempts to call {{.get()}} on a Scala {{Option}} that contains {{{}null{}}}, then immediately invokes {{.map()}} on the null result. This violates Scala's Option contract and causes the NullPointerException.
> h3. Error Flow
>  # The streaming query completes a micro-batch trigger
>  # Progress reporting attempts to extract source metrics via {{ProgressContext.extractSourceProgress()}}
>  # The KafkaMicroBatchStream is queried for its metrics
>  # An unsafe {{.get()}} call on an empty or null Option returns null
>  # Attempting to call {{.map()}} on the null value throws NullPointerException
> h3. Impact
>  * The streaming query crashes and stops processing
>  * Progress information cannot be generated for the current batch
>  * The failure occurs during the finishTrigger phase, after batch processing completes
> h3. Technical Details
>  * {*}Component{*}: Spark Structured Streaming with Kafka source
>  * {*}Class{*}: {{org.apache.spark.sql.kafka010.KafkaMicroBatchStream}}
>  * {*}Method{*}: {{metrics()}} at line 520
>  * {*}Trigger Type{*}: ProcessingTime (micro-batch mode)
> h3. Likely Causes
>  * Missing or unavailable Kafka metrics configuration
>  * Empty/uninitialized metrics collection in the Kafka source
>  * Race condition where metrics are accessed before initialization
>  * Incompatibility between Spark and Kafka connector versions



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org