You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/09/08 07:20:28 UTC
[PR] [SPARK-59308][ML] Avoid integer overflow when computing ML feature and image sizes [spark]
HyukjinKwon opened a new pull request, #58596:
URL: https://github.com/apache/spark/pull/58596
### What changes were proposed in this pull request?
Two size computations that can overflow a 32-bit `Int` are made overflow-safe:
- `ImageSchema.decode` computes the decoded byte size as `height * width * nChannels`. This is now
computed in `Long` so the multiplication cannot wrap around and slip past the existing
`< 1e9` size assertion (which would then allocate a wrongly-sized or negative array).
- `Interaction` computes the output-vector size as a product of per-feature output sizes with
`size *= currentEncoder.outputSize`. This now uses `Math.multiplyExact` so that a product
exceeding `Int.MaxValue` fails with a clear arithmetic error instead of wrapping to a bogus
(possibly negative) size and producing corrupt indices.
### Why are the changes needed?
Both products can exceed `Int.MaxValue` for large-but-representable inputs, and silently wrapping
produces a wrong or negative allocation size rather than a clear failure. A legitimate image
(< 1e9 bytes) and a legitimate feature interaction (which must fit in an `Int`-indexed vector) are
unaffected.
### Does this PR introduce _any_ user-facing change?
No, other than a clear error instead of a corrupt result / wrong-size allocation on inputs whose
size would overflow `Int`.
### How was this patch tested?
Existing `ImageSchema`/`Interaction` tests. The overflow branches are not unit-tested because
triggering them requires inputs large enough (a multi-billion-pixel image; nominal attribute
counts whose product exceeds `Int.MaxValue`) to exhaust test heap before the check runs; the
changes are local (`Long` arithmetic / `Math.multiplyExact`).
### Was this patch authored or co-authored using generative AI tooling?
Generated-by: Isaac
This pull request and its description were written by Isaac.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #58596: [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size
URL: https://github.com/apache/spark/pull/58596
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size [spark]
Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #58596:
URL: https://github.com/apache/spark/pull/58596#issuecomment-5593306388
Merged to master, branch-4.x and branch-4.3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59308][ML] Avoid integer overflow when computing ML feature and image sizes [spark]
Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58596:
URL: https://github.com/apache/spark/pull/58596#issuecomment-5581054961
The two changes here have quite different characters: the `ImageSchema` one looks good, but the
`Interaction` one guards a path that I believe cannot be reached.
### 1. `ImageSchema.decode` — LGTM
```scala
val imageSize = height.toLong * width * nChannels
assert(imageSize < 1e9, "image is too large")
val decoded = Array.ofDim[Byte](imageSize.toInt)
```
The widening is correct (`Long * Int * Int` stays `Long`, the `Long < Double` comparison is fine,
and `toInt` is safe once the assertion has passed), and the overflow is genuinely reachable: a
`BufferedImage` must have an `Int`-representable pixel count, but e.g. an 800M-pixel RGB image gives
`8e8 * 3 = 2.4e9`, which wrapped negative, slipped past the assertion, and produced a
`NegativeArraySizeException`. Now it fails with the intended "image is too large". I also checked
that this is the only place in `mllib` doing an `nChannels` multiplication, so nothing is missed.
### 2. `Interaction` — the overflow this guards can't happen
`size *= currentEncoder.outputSize` runs inside the UDF, i.e. on the executor. But before any of
that, `transform` eagerly calls `getFeatureAttrs` on the driver:
https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/Interaction.scala#L75
and that method builds the full cross product of attributes via the nested `flatMap` at L174-L181,
then `toArray` at L184 and `toMetadata()` at L116. The number of attributes it materializes is
*exactly* the same product as `size` — for each input feature, the length of `encodedFeatureAttrs`
equals that feature's `FeatureEncoder.outputSize`.
So to get `size` anywhere near `Int.MaxValue`, the driver first has to materialize ~2 billion
`Attribute` objects and serialize them into column metadata. It will OOM long before the UDF ever
runs. There is no input that reaches the `Math.multiplyExact` call.
A few consequences:
- The PR description's failure mode — "wrapping to a bogus (possibly negative) size and producing
corrupt indices" — can't actually occur, so as written the description records a bug that isn't
there.
- "How was this patch tested?" is right that the heap runs out first, but for a different reason:
it's the driver-side attribute cross product, not the size of the input data.
- If fail-fast on an oversized interaction is the goal, the check belongs on the driver right
before L75 — computing the product of `featureEncoders.map(_.outputSize)` in `Long` and rejecting
it there would fail with a clear message instead of OOM-ing while building billions of attributes.
That would be a change users can actually observe.
Also, even keeping the current placement, `Math.multiplyExact` throws
`ArithmeticException("integer overflow")`, which then gets wrapped in
`FAILED_EXECUTE_USER_DEFINED_FUNCTION`. That's not really the "clear arithmetic error" the
description promises; a `require` with an explanatory message would read better.
### 3. Comment size
Four lines of comment for a one-line change in `Interaction` (and three in `ImageSchema`) is heavy
relative to the surrounding code. `Math.multiplyExact` / `.toLong` already state the intent; a
single line on *why* overflow matters here should be enough.
### 4. Minor, out of scope
`FeatureEncoder.outputSize = numFeatures.sum` can overflow `Int` in principle too, but only for an
unrealistically large `numFeatures` array. Not worth touching in this PR.
### Summary
| | |
|---|---|
| `ImageSchema.decode` widening | LGTM — correct, and fixes a reachable path |
| `Interaction` `multiplyExact` | Please reconsider — unreachable as placed; either move the check to the driver at L75 or drop it from this PR |
| PR description | The `Interaction` parts (corrupt indices / why it isn't testable) need correcting |
| Comments | Suggest trimming |
No build or lint concerns: types are correct, no non-ASCII, no lines over 100 chars. Dropping to
just the `ImageSchema` change would be mergeable as is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org