You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "giriprakash (via GitHub)" <gi...@apache.org> on 2026/08/10 10:56:50 UTC
[PR] [SPARK-41246][CORE] Parse RDD block ids whose RDD id overflowed to negative [spark]
giriprakash opened a new pull request, #57893:
URL: https://github.com/apache/spark/pull/57893
### What changes were proposed in this pull request?
Allow `BlockId.apply` to parse RDD block names whose RDD id is negative, e.g. `rdd_-1330910599_36`.
`RDDBlockId.name` already renders negative ids with a leading minus sign. The only change is to make the RDD id regex accept an optional minus so the name round-trips through `BlockId.apply`. The split index stays non-negative, and no other block id pattern is changed.
### Why are the changes needed?
`SparkContext` hands out RDD ids from a 32-bit `AtomicInteger` that never resets while the application runs. After `Int.MaxValue` RDDs, the counter wraps to negative values. Cached blocks then get names such as `rdd_-1330910599_36`. Paths that reconstruct a `BlockId` from its name — including `UpdateBlockInfo.readExternal` — failed with `UnrecognizedBlockId` because the parser only accepted digits. The application stayed up, but queries kept failing until restart.
This matches the failure mode described in [SPARK-41246](https://issues.apache.org/jira/browse/SPARK-41246).
### Does this PR introduce _any_ user-facing change?
Yes. Applications that previously failed with `UnrecognizedBlockId` after the RDD id counter overflowed can now parse and use those block ids. Behavior for all previously valid (non-negative) RDD block ids is unchanged.
### How was this patch tested?
Added a unit test in `BlockIdSuite` that:
- parses the reported failing name `rdd_-1330910599_36`
- parses `rdd_-2147483648_0` (`Int.MinValue`)
- confirms positive ids still round-trip and remain distinct from their negative counterparts
Please run:
```
build/sbt "core/testOnly org.apache.spark.storage.BlockIdSuite"
```
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Cursor Agent
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on PR #57893:
URL: https://github.com/apache/spark/pull/57893#issuecomment-5260455079
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/3df71625ce243d4f76099cab3d4fa7b52f13006c
- merged into branch-4.x https://github.com/apache/spark/commit/933a9caf0114cb9fa9840ba4f7a97d0c7fa1a080
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan commented on code in PR #57893:
URL: https://github.com/apache/spark/pull/57893#discussion_r3759061542
##########
core/src/main/scala/org/apache/spark/storage/BlockId.scala:
##########
@@ -263,7 +263,11 @@ case class CacheId(sessionUUID: String, hash: String) extends BlockId {
@DeveloperApi
object BlockId {
- val RDD = "rdd_([0-9]+)_([0-9]+)".r
+ // Safety net: newRddId() fail-fasts before minting negative ids, but names like
Review Comment:
**Nit:**
```suggestion
// Safety net: newRddId() fails fast before minting negative ids, but names like
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net [spark]
Posted by "giriprakash (via GitHub)" <gi...@apache.org>.
giriprakash commented on code in PR #57893:
URL: https://github.com/apache/spark/pull/57893#discussion_r3759197145
##########
core/src/main/scala/org/apache/spark/storage/BlockId.scala:
##########
@@ -263,7 +263,11 @@ case class CacheId(sessionUUID: String, hash: String) extends BlockId {
@DeveloperApi
object BlockId {
- val RDD = "rdd_([0-9]+)_([0-9]+)".r
+ // Safety net: newRddId() fail-fasts before minting negative ids, but names like
Review Comment:
Done, Thanks!!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net [spark]
Posted by "cloud-fan (via GitHub)" <gi...@apache.org>.
cloud-fan closed pull request #57893: [SPARK-41246][CORE] Fail fast on RDD id overflow; parse negative block ids as safety net
URL: https://github.com/apache/spark/pull/57893
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-41246][CORE] Parse RDD block ids whose RDD id overflowed to negative [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57893:
URL: https://github.com/apache/spark/pull/57893#issuecomment-5243154828
Thank you @giriprakash!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org