You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HyukjinKwon (via GitHub)" <gi...@apache.org> on 2026/09/08 07:20:28 UTC

[PR] [SPARK-59308][ML] Avoid integer overflow when computing ML feature and image sizes [spark]

HyukjinKwon opened a new pull request, #58596:
URL: https://github.com/apache/spark/pull/58596

   ### What changes were proposed in this pull request?
   
   Two size computations that can overflow a 32-bit `Int` are made overflow-safe:
   - `ImageSchema.decode` computes the decoded byte size as `height * width * nChannels`. This is now
     computed in `Long` so the multiplication cannot wrap around and slip past the existing
     `< 1e9` size assertion (which would then allocate a wrongly-sized or negative array).
   - `Interaction` computes the output-vector size as a product of per-feature output sizes with
     `size *= currentEncoder.outputSize`. This now uses `Math.multiplyExact` so that a product
     exceeding `Int.MaxValue` fails with a clear arithmetic error instead of wrapping to a bogus
     (possibly negative) size and producing corrupt indices.
   
   ### Why are the changes needed?
   
   Both products can exceed `Int.MaxValue` for large-but-representable inputs, and silently wrapping
   produces a wrong or negative allocation size rather than a clear failure. A legitimate image
   (< 1e9 bytes) and a legitimate feature interaction (which must fit in an `Int`-indexed vector) are
   unaffected.
   
   ### Does this PR introduce _any_ user-facing change?
   
   No, other than a clear error instead of a corrupt result / wrong-size allocation on inputs whose
   size would overflow `Int`.
   
   ### How was this patch tested?
   
   Existing `ImageSchema`/`Interaction` tests. The overflow branches are not unit-tested because
   triggering them requires inputs large enough (a multi-billion-pixel image; nominal attribute
   counts whose product exceeds `Int.MaxValue`) to exhaust test heap before the check runs; the
   changes are local (`Long` arithmetic / `Math.multiplyExact`).
   
   ### Was this patch authored or co-authored using generative AI tooling?
   
   Generated-by: Isaac
   
   This pull request and its description were written by Isaac.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon closed pull request #58596: [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size
URL: https://github.com/apache/spark/pull/58596


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59308][ML] Avoid Int overflow when computing the decoded image size [spark]

Posted by "HyukjinKwon (via GitHub)" <gi...@apache.org>.
HyukjinKwon commented on PR #58596:
URL: https://github.com/apache/spark/pull/58596#issuecomment-5593306388

   Merged to master, branch-4.x and branch-4.3.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


Re: [PR] [SPARK-59308][ML] Avoid integer overflow when computing ML feature and image sizes [spark]

Posted by "dongjoon-hyun (via GitHub)" <gi...@apache.org>.
dongjoon-hyun commented on PR #58596:
URL: https://github.com/apache/spark/pull/58596#issuecomment-5581054961

   The two changes here have quite different characters: the `ImageSchema` one looks good, but the
   `Interaction` one guards a path that I believe cannot be reached.
   
   ### 1. `ImageSchema.decode` — LGTM
   
   ```scala
   val imageSize = height.toLong * width * nChannels
   assert(imageSize < 1e9, "image is too large")
   val decoded = Array.ofDim[Byte](imageSize.toInt)
   ```
   
   The widening is correct (`Long * Int * Int` stays `Long`, the `Long < Double` comparison is fine,
   and `toInt` is safe once the assertion has passed), and the overflow is genuinely reachable: a
   `BufferedImage` must have an `Int`-representable pixel count, but e.g. an 800M-pixel RGB image gives
   `8e8 * 3 = 2.4e9`, which wrapped negative, slipped past the assertion, and produced a
   `NegativeArraySizeException`. Now it fails with the intended "image is too large". I also checked
   that this is the only place in `mllib` doing an `nChannels` multiplication, so nothing is missed.
   
   ### 2. `Interaction` — the overflow this guards can't happen
   
   `size *= currentEncoder.outputSize` runs inside the UDF, i.e. on the executor. But before any of
   that, `transform` eagerly calls `getFeatureAttrs` on the driver:
   
   https://github.com/apache/spark/blob/master/mllib/src/main/scala/org/apache/spark/ml/feature/Interaction.scala#L75
   
   and that method builds the full cross product of attributes via the nested `flatMap` at L174-L181,
   then `toArray` at L184 and `toMetadata()` at L116. The number of attributes it materializes is
   *exactly* the same product as `size` — for each input feature, the length of `encodedFeatureAttrs`
   equals that feature's `FeatureEncoder.outputSize`.
   
   So to get `size` anywhere near `Int.MaxValue`, the driver first has to materialize ~2 billion
   `Attribute` objects and serialize them into column metadata. It will OOM long before the UDF ever
   runs. There is no input that reaches the `Math.multiplyExact` call.
   
   A few consequences:
   
   - The PR description's failure mode — "wrapping to a bogus (possibly negative) size and producing
     corrupt indices" — can't actually occur, so as written the description records a bug that isn't
     there.
   - "How was this patch tested?" is right that the heap runs out first, but for a different reason:
     it's the driver-side attribute cross product, not the size of the input data.
   - If fail-fast on an oversized interaction is the goal, the check belongs on the driver right
     before L75 — computing the product of `featureEncoders.map(_.outputSize)` in `Long` and rejecting
     it there would fail with a clear message instead of OOM-ing while building billions of attributes.
     That would be a change users can actually observe.
   
   Also, even keeping the current placement, `Math.multiplyExact` throws
   `ArithmeticException("integer overflow")`, which then gets wrapped in
   `FAILED_EXECUTE_USER_DEFINED_FUNCTION`. That's not really the "clear arithmetic error" the
   description promises; a `require` with an explanatory message would read better.
   
   ### 3. Comment size
   
   Four lines of comment for a one-line change in `Interaction` (and three in `ImageSchema`) is heavy
   relative to the surrounding code. `Math.multiplyExact` / `.toLong` already state the intent; a
   single line on *why* overflow matters here should be enough.
   
   ### 4. Minor, out of scope
   
   `FeatureEncoder.outputSize = numFeatures.sum` can overflow `Int` in principle too, but only for an
   unrealistically large `numFeatures` array. Not worth touching in this PR.
   
   ### Summary
   
   | | |
   |---|---|
   | `ImageSchema.decode` widening | LGTM — correct, and fixes a reachable path |
   | `Interaction` `multiplyExact` | Please reconsider — unreachable as placed; either move the check to the driver at L75 or drop it from this PR |
   | PR description | The `Interaction` parts (corrupt indices / why it isn't testable) need correcting |
   | Comments | Suggest trimming |
   
   No build or lint concerns: types are correct, no non-ASCII, no lines over 100 chars. Dropping to
   just the `ImageSchema` change would be mergeable as is.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org