You are viewing a plain text version of this content. The canonical link for it is here.

Posted to reviews@spark.apache.org by "dongjoon-hyun (via GitHub)" <gi...@apache.org> on 2026/06/15 19:39:30 UTC

[PR] [SPARK-57475] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

dongjoon-hyun opened a new pull request, #56523:
URL: https://github.com/apache/spark/pull/56523

### What changes were proposed in this pull request?

`Encoders.BitmapArrays.decode()` now validates the count prefix before allocating the array via `Objects.checkFromIndexSize(0, numBitmaps, buf.readableBytes() / 8)`, which requires `0 <= numBitmaps <= buf.readableBytes() / 8` and throws `IndexOutOfBoundsException` otherwise. The divisor `8` is the minimum on-wire size of one element, since an empty `RoaringBitmap` serializes to 8 bytes (a 4-byte cookie followed by a 4-byte size).

### Why are the changes needed?

`decode()` allocated `new RoaringBitmap[numBitmaps]` from an untrusted count.
- A negative value throws an opaque `NegativeArraySizeException`.
- An oversized value can trigger `OutOfMemoryError` on a corrupt or hostile frame.

Validating first fails fast with a clear error.

### Does this PR introduce _any_ user-facing change?

No.

### How was this patch tested?

Pass the CIs with newly added test cases.

### Was this patch authored or co-authored using generative AI tooling?

Generated-by: Claude Code (Claude Opus 4.8)

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712829764

   Thank you so much, @sarutak !


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712429556

   Could you review this PR when you have some time, @gengliangwang ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712677314

   Could you review this when you have some time, @sarutak ?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4711869273

   Note that this is a kind of robustness improvement unlike the other relevant PRs.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712905012

   Thank you, @HyukjinKwon , too.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712675451

   `core` module passed the CIs.
   
   <img width="412" height="110" alt="Screenshot 2026-06-15 at 14 39 31" src="https://github.com/user-attachments/assets/ed1b3fc2-4f11-4473-8907-b7d646b9d287" />
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun commented on PR #56523:
URL: https://github.com/apache/spark/pull/56523#issuecomment-4712914141

   Merged to master/4.x for Apache Spark 4.3.0.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

Re: [PR] [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode` [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.

dongjoon-hyun closed pull request #56523: [SPARK-57475][CORE] Validate length before allocation in `Encoders.BitmapArrays.decode`
URL: https://github.com/apache/spark/pull/56523


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org