You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by vi...@apache.org on 2026/06/24 14:15:42 UTC
(spark) branch branch-4.2 updated: [SPARK-57438][SS] Fix NullPointerException in Kafka source metrics when latest partition offsets are unavailable
This is an automated email from the ASF dual-hosted git repository.
viirya pushed a commit to branch branch-4.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-4.2 by this push:
new 3fa6f9f71493 [SPARK-57438][SS] Fix NullPointerException in Kafka source metrics when latest partition offsets are unavailable
3fa6f9f71493 is described below
commit 3fa6f9f714939a5962143f8bc843b3e1907fedf7
Author: Anupam Yadav <an...@amazon.com>
AuthorDate: Wed Jun 24 07:15:07 2026 -0700
[SPARK-57438][SS] Fix NullPointerException in Kafka source metrics when latest partition offsets are unavailable
### What changes were proposed in this pull request?
Fix a `NullPointerException` in the Kafka micro-batch source when custom metrics are reported before the latest partition offsets are known.
In the non-real-time-mode branch of the instance `metrics()`, `latestPartitionOffsets` (a `var` initialized to null until `latestOffset()` runs) was wrapped as `Some(latestPartitionOffsets)`, producing `Some(null)`. The companion `metrics(...)` then evaluated `isDefined` as true, called `.get` (null), and invoked `.map` on it, throwing. The fix changes that single wrap to `Option(latestPartitionOffsets)`, so a null becomes `None` and the metrics computation is correctly skipped.
### Why are the changes needed?
`metrics()` can be called before the first batch completes (e.g. progress reporting), while `latestPartitionOffsets` is still null:
```
java.lang.NullPointerException: Cannot invoke "scala.collection.IterableOps.map(scala.Function1)"
because the return value of "scala.Option.get()" is null
at org.apache.spark.sql.kafka010.KafkaMicroBatchStream$.metrics(KafkaMicroBatchStream.scala:520)
```
This was introduced by #52729 (SPARK-54027, Kafka RTM support).
### Does this PR introduce _any_ user-facing change?
No behavioral change for well-formed inputs. It prevents the streaming query from crashing when metrics are requested before the latest partition offsets are available; in that case an empty metrics map is returned, as intended.
### How was this patch tested?
Added a test in `KafkaMicroBatchSourceSuite` that drives the instance `metrics()` while `latestPartitionOffsets` is still null (the stream is built but `latestOffset()` has not run), asserting an empty map and no NPE. Verified the test fails with the original NPE if the call-site fix is reverted to `Some(...)`.
### Was this patch authored or co-authored using generative AI tooling?
Yes.
Closes #56526 from yadavay-amzn/fix/SPARK-57438-kafka-metrics-npe.
Authored-by: Anupam Yadav <an...@amazon.com>
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
(cherry picked from commit 1125957250c928167516a429efaf7dcb231a141c)
Signed-off-by: Liang-Chi Hsieh <vi...@gmail.com>
---
.../spark/sql/kafka010/KafkaMicroBatchStream.scala | 2 +-
.../sql/kafka010/KafkaMicroBatchSourceSuite.scala | 26 ++++++++++++++++++++++
2 files changed, 27 insertions(+), 1 deletion(-)
diff --git a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
index 828891f0b498..17323165b451 100644
--- a/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
+++ b/connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchStream.scala
@@ -357,7 +357,7 @@ private[kafka010] class KafkaMicroBatchStream(
}
}
} else {
- Some(latestPartitionOffsets)
+ Option(latestPartitionOffsets)
}
KafkaMicroBatchStream.metrics(latestConsumedOffset, reCalculatedLatestPartitionOffsets)
diff --git a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
index e6a794c22e1f..274be623da6f 100644
--- a/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
+++ b/connector/kafka-0-10-sql/src/test/scala/org/apache/spark/sql/kafka010/KafkaMicroBatchSourceSuite.scala
@@ -1883,6 +1883,32 @@ abstract class KafkaMicroBatchV2SourceSuite extends KafkaMicroBatchSourceSuiteBa
// test null latestAvailablePartitionOffsets
assert(KafkaMicroBatchStream.metrics(Optional.ofNullable(offset), None).isEmpty)
}
+
+ test("SPARK-57438: metrics should not NPE when latestPartitionOffsets is null") {
+ // Construct a KafkaMicroBatchStream instance without calling latestOffset(),
+ // so latestPartitionOffsets remains null (its default uninitialized value).
+ // Calling metrics() on this instance exercises the real non-RTM code path.
+ val topic = newTopic()
+ val tp = new TopicPartition(topic, 0)
+
+ SparkSession.setActiveSession(spark)
+ withTempDir { dir =>
+ val provider = new KafkaSourceProvider()
+ val options = Map(
+ "kafka.bootstrap.servers" -> testUtils.brokerAddress,
+ "subscribe" -> topic
+ )
+ val dsOptions = new CaseInsensitiveStringMap(options.asJava)
+ val table = provider.getTable(dsOptions)
+ val stream = table.newScanBuilder(dsOptions).build().toMicroBatchStream(dir.getAbsolutePath)
+ .asInstanceOf[KafkaMicroBatchStream]
+
+ // latestPartitionOffsets is still null - metrics() must not NPE
+ val offset = KafkaSourceOffset(Map(tp -> 0L))
+ val result = stream.metrics(Optional.of(offset))
+ assert(result.isEmpty)
+ }
+ }
}
class KafkaMicroBatchV1SourceWithAdminSuite extends KafkaMicroBatchV1SourceSuite {
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org