You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "leixm (via GitHub)" <gi...@apache.org> on 2026/09/10 11:14:06 UTC
[PR] [SPARK-59403][CORE] Fix remote fetch of empty deserialized RDD blocks [spark]
leixm opened a new pull request, #58697:
URL: https://github.com/apache/spark/pull/58697
### What changes were proposed in this pull request?
This PR associates the representative `BlockStatus` returned by
`BlockLocationsAndStatus` with the `BlockManagerId` that owns it.
`BlockManager` now accepts an empty remote response only when it is fetched
from that exact location and the location reports a deserialized in-memory
block. Empty responses from external shuffle services or other locations
continue to be treated as fetch failures.
The PR also adds regression tests for:
- accepting an empty response from the matching deserialized-memory executor;
- rejecting an empty external shuffle service response; and
- rejecting an empty response from an executor other than the status owner.
### Why are the changes needed?
For deserialized in-memory blocks, `memSize` is an estimated size of JVM
objects, not the size of the serialized data sent over the network. Therefore,
an empty iterator can have a positive `memSize` while its serialized response
is empty.
The previous check interpreted every empty response for a positive block size
as a failed fetch. A simple exception based only on the representative status
would be unsafe because the status can belong to a different replica. This PR
keeps the corruption and stale-location protection by applying the exception
only to the status-owning executor.
### Does this PR introduce _any_ user-facing change?
Yes. Remote reads of an affected empty deserialized RDD cache block can now
succeed instead of being incorrectly reported as unavailable. There are no
public API or configuration changes.
### How was this patch tested?
```bash
build/sbt 'core/testOnly org.apache.spark.storage.BlockManagerSuite'
The suite completed successfully with 133 tests passed.
###Was this patch authored or co-authored using generative AI tooling?
Generated-by: OpenAI Codex 5.6 Terra.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59403][CORE] Fix remote fetch of empty deserialized RDD blocks [spark]
Posted by "leixm (via GitHub)" <gi...@apache.org>.
leixm commented on PR #58697:
URL: https://github.com/apache/spark/pull/58697#issuecomment-5630625332
cc @bozhang2820 @AngersZhuuuu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org