You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "lnagel (via GitHub)" <gi...@apache.org> on 2026/04/07 11:13:48 UTC
[I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
lnagel opened a new issue, #55236:
URL: https://github.com/apache/spark/issues/55236
Project: SPARK
Type: Bug
Component: Structured Streaming, Kafka
Affects Version: 4.1.1
Priority: Critical
KafkaMicroBatchStream.metrics() throws a NullPointerException when called before latestOffset() has populated the latestPartitionOffsets field.
This is a regression introduced in SPARK-54027 (PR #52729), which refactored the metrics method to wrap latestPartitionOffsets in Option. The original implementation (from SPARK-34854) passed the value directly and used an explicit null check, which was safe:
```
// Before SPARK-54027 (safe)
if (offset.nonEmpty && latestAvailablePartitionOffsets != null) {
```
After SPARK-54027, the instance method wraps the value in Some():
```scala
// KafkaMicroBatchStream.scala, instance metrics method
} else {
Some(latestPartitionOffsets) // creates Some(null) when field is uninitialized
}
```
And the companion object method uses .isDefined:
```scala
// KafkaMicroBatchStream.scala, companion object metrics method
if (offset.nonEmpty && latestAvailablePartitionOffsets.isDefined) {
...
val offsetsBehindLatest = latestAvailablePartitionOffsets.get // returns null
.map(...) // NullPointerException
```
Since latestPartitionOffsets is declared as `private var latestPartitionOffsets: PartitionOffsetMap = _` (null), wrapping it in Some() before latestOffset() runs creates Some(null). The .isDefined check passes (Some(null).isDefined == true), then .get returns null, and calling .map() on null throws the NPE.
This crashes the streaming query during finishTrigger -> extractSourceProgress -> metrics, with no way to recover.
Stack trace:
```
Caused by: java.lang.NullPointerException: Cannot invoke "scala.collection.IterableOps.map(scala.Function1)" because the return value of "scala.Option.get()" is null
at org.apache.spark.sql.kafka010.KafkaMicroBatchStream$.metrics(KafkaMicroBatchStream.scala:520)
at org.apache.spark.sql.kafka010.KafkaMicroBatchStream.metrics(KafkaMicroBatchStream.scala:363)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.$anonfun$extractSourceProgress$2(ProgressReporter.scala:384)
at org.apache.spark.util.Utils$.timeTakenMs(Utils.scala:496)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.$anonfun$extractSourceProgress$1(ProgressReporter.scala:380)
at scala.collection.immutable.List.map(List.scala:236)
at scala.collection.immutable.List.map(List.scala:79)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.extractSourceProgress(ProgressReporter.scala:379)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.constructNewProgress(ProgressReporter.scala:348)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.finishTrigger(ProgressReporter.scala:312)
at org.apache.spark.sql.execution.streaming.runtime.ProgressContext.finishTrigger(ProgressReporter.scala:429)
at org.apache.spark.sql.execution.streaming.runtime.MicroBatchExecution.executeOneBatch(MicroBatchExecution.scala:525)
```
Fix: change `Some(latestPartitionOffsets)` to `Option(latestPartitionOffsets)` in the instance metrics method, which converts null to None instead of Some(null).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "isaias-b (via GitHub)" <gi...@apache.org>.
isaias-b commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-5729834897
Maybe closed because its fixed in later version?
https://issues.apache.org/jira/browse/SPARK-55271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "lnagel (via GitHub)" <gi...@apache.org>.
lnagel commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-5602097893
A bot closed it without any meaningful actions taken.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "gaogaotiantian (via GitHub)" <gi...@apache.org>.
gaogaotiantian commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-4201182999
@jerrypeng could you take a look as the original author? Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "k3lyan (via GitHub)" <gi...@apache.org>.
k3lyan commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-5729649870
Hi, we encounter the same issue in our project.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "isaias-b (via GitHub)" <gi...@apache.org>.
isaias-b commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-5729708812
> [@jerrypeng](https://github.com/jerrypeng), [@gaogaotiantian](https://github.com/gaogaotiantian), [@lnagel](https://github.com/lnagel) - could you clarify why this issue was closed? Has it been resolved, or is there a workaround?
Would be interested in a workaround until it's fixed.
> A bot closed it without any meaningful actions taken.
Will it be reopened and fixed actually?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "ntnhaatj (via GitHub)" <gi...@apache.org>.
ntnhaatj commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-4581966969
hi, I encountered the same
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "0xdarkman (via GitHub)" <gi...@apache.org>.
0xdarkman commented on issue #55236:
URL: https://github.com/apache/spark/issues/55236#issuecomment-5600456766
@jerrypeng, @gaogaotiantian, @lnagel - could you clarify why this issue was closed? Has it been resolved, or is there a workaround?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [I] NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #55236: NullPointerException in KafkaMicroBatchStream.metrics when latestPartitionOffsets is uninitialized
URL: https://github.com/apache/spark/issues/55236
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org