You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "superdiaodiao (via GitHub)" <gi...@apache.org> on 2026/07/16 08:18:15 UTC
[PR] [SPARK-49123][SS] Log empty Kafka polls with advanced offsets [spark]
superdiaodiao opened a new pull request, #57297:
URL: https://github.com/apache/spark/pull/57297
### What changes were proposed in this pull request?
This PR updates `KafkaDataConsumer` to log a warning when a Kafka poll returns zero visible records but advances the consumer position. It also fixes the nearby comment from "not invisible" to "invisible".
### Why are the changes needed?
When transactional or aborted Kafka messages are invisible under `isolation.level=read_committed`, Spark can poll zero visible records while the offset still advances. This is a valid path, so Spark should not throw, but the current behavior is silent and makes troubleshooting poll delays difficult.
The warning gives users the group id, topic partition, poll timeout, and offset movement so they can distinguish this path from out-of-range offsets and true poll timeouts.
### Does this PR introduce _any_ user-facing change?
Yes. Spark now emits a warning for the valid empty-poll path where the Kafka consumer position advances despite no visible records being returned.
### How was this patch tested?
- `git diff --check`
- `python3 dev/structured_logging_style.py --help` (the script completed and reported the structured logging style check passed)
Not run locally:
- `./dev/scalafmt --test` was attempted, but the script maps to `./build/mvn scalafmt:format ...` and produced no output for over 90 seconds, so I interrupted it.
- `./build/mvn -pl connector/kafka-0-10-sql -DskipTests compile` was attempted, but produced no output for over 60 seconds, so I interrupted it.
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Codex
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49123][SS] Log empty Kafka polls with advanced offsets [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on code in PR #57297:
URL: https://github.com/apache/spark/pull/57297#discussion_r3595512818
##########
connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala:
##########
@@ -97,14 +97,21 @@ private[kafka010] class InternalKafkaConsumer(
// - `offset` is out of range so that Kafka returns nothing. `OffsetOutOfRangeException` will
// be thrown.
// - Cannot fetch any data before timeout. `TimeoutException` will be thrown.
- // - Fetched something but all of them are not invisible. This is a valid case and let the
- // caller handles this.
+ // - Fetched something but all of them are invisible. This is a valid case and let the
+ // caller handle this.
if (offset < range.earliest || offset >= range.latest) {
throw new OffsetOutOfRangeException(
Map(topicPartition -> java.lang.Long.valueOf(offset)).asJava)
} else if (offset == offsetAfterPoll) {
throw new TimeoutException(
s"Cannot fetch record for offset $offset in $pollTimeoutMs milliseconds")
+ } else {
+ logWarning(log"Polled zero visible records for ${MDC(TOPIC_PARTITION, topicPartition)} " +
Review Comment:
too verbose? WARNING is not a suitable choice because it's a normal case and do not expect the user to take action, DEBUG might be more appropriate. INFO is also ok if it's rare to go to this branch in real workloads
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49123][SS] Log empty Kafka polls with advanced offsets [spark]
Posted by "superdiaodiao (via GitHub)" <gi...@apache.org>.
superdiaodiao commented on code in PR #57297:
URL: https://github.com/apache/spark/pull/57297#discussion_r3595808138
##########
connector/kafka-0-10-sql/src/main/scala/org/apache/spark/sql/kafka010/consumer/KafkaDataConsumer.scala:
##########
@@ -97,14 +97,21 @@ private[kafka010] class InternalKafkaConsumer(
// - `offset` is out of range so that Kafka returns nothing. `OffsetOutOfRangeException` will
// be thrown.
// - Cannot fetch any data before timeout. `TimeoutException` will be thrown.
- // - Fetched something but all of them are not invisible. This is a valid case and let the
- // caller handles this.
+ // - Fetched something but all of them are invisible. This is a valid case and let the
+ // caller handle this.
if (offset < range.earliest || offset >= range.latest) {
throw new OffsetOutOfRangeException(
Map(topicPartition -> java.lang.Long.valueOf(offset)).asJava)
} else if (offset == offsetAfterPoll) {
throw new TimeoutException(
s"Cannot fetch record for offset $offset in $pollTimeoutMs milliseconds")
+ } else {
+ logWarning(log"Polled zero visible records for ${MDC(TOPIC_PARTITION, topicPartition)} " +
Review Comment:
Thanks for the review. That makes sense for this valid path; I lowered it to DEBUG and updated the PR description accordingly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49123][SS] Log empty Kafka polls with advanced offsets [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 commented on PR #57297:
URL: https://github.com/apache/spark/pull/57297#issuecomment-5001743428
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/fe0e50253e3d72f9317582659e096c36230b9ae9
- merged into branch-4.x https://github.com/apache/spark/commit/c081674a80417fec7d78846b51c4d0780e7bf54f
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-49123][SS] Log empty Kafka polls with advanced offsets [spark]
Posted by "pan3793 (via GitHub)" <gi...@apache.org>.
pan3793 closed pull request #57297: [SPARK-49123][SS] Log empty Kafka polls with advanced offsets
URL: https://github.com/apache/spark/pull/57297
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org