You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "moomindani (via GitHub)" <gi...@apache.org> on 2026/03/09 08:29:03 UTC
[PR] [SPARK-55523][SQL] Fix inputMetrics.recordsRead to report rows instead of CachedBatches [spark]
moomindani opened a new pull request, #54684:
URL: https://github.com/apache/spark/pull/54684
### What changes were proposed in this pull request?
`InMemoryTableScanExec.doExecute()` iterates over `CachedBatch` objects backed by `RDD.getOrCompute`, which increments `inputMetrics.recordsRead` by 1 per `CachedBatch` on a cache hit — counting batches instead of actual rows.
This fix registers a `TaskCompletionListener` inside `mapPartitionsInternal` that corrects the count once all batches in a partition are consumed:
- **Cache hit:** `getOrCompute` added `numBatches`; the listener adds `(rowCount - numBatches)` so the final value equals the actual row count.
- **Cache miss:** the source scan already incremented `recordsRead` by `rowCount`; the listener adds `(rowCount - rowCount) = 0`, leaving the value unchanged.
No changes to `RDD.scala`, so non-SQL RDD caching metrics are unaffected.
### Why are the changes needed?
Users and monitoring tools relying on `TaskMetrics.inputMetrics.recordsRead` (e.g. the Spark UI, external observability systems) see incorrect values when reading from a cached DataFrame. For example, 3,605 rows stored in a single `CachedBatch` would report `recordsRead = 1` instead of `3605`.
### Does this PR introduce _any_ user-facing change?
Yes — `inputMetrics.recordsRead` for queries reading from a cached DataFrame now reports the correct row count instead of the batch count.
### How was this patch tested?
Added a test in `InMemoryColumnarQuerySuite` that caches 1000 rows, materializes the cache, then reads from cache and asserts `inputMetrics.recordsRead == 1000` via a `SparkListener`.
### Notes
`doExecuteColumnar()` has the same underlying issue (batch-level counting via `getOrCompute`) and could be addressed as a follow-up.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55523][SQL] Fix inputMetrics.recordsRead to report rows instead of CachedBatches [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #54684: [SPARK-55523][SQL] Fix inputMetrics.recordsRead to report rows instead of CachedBatches
URL: https://github.com/apache/spark/pull/54684
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-55523][SQL] Fix inputMetrics.recordsRead to report rows instead of CachedBatches [spark]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #54684:
URL: https://github.com/apache/spark/pull/54684#issuecomment-4736911268
We're closing this PR because it hasn't been updated in a while. This isn't a judgement on the merit of the PR in any way. It's just a way of keeping the PR queue manageable.
If you'd like to revive this PR, please reopen it and ask a committer to remove the Stale tag!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org