[jira] [Commented] (HADOOP-18896) NegativeArraySizeException thrown in FSOutputSummer.java given large file.bytes-per-checksum

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HADOOP-18896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=18101780#comment-18101780 ] 

ASF GitHub Bot commented on HADOOP-18896:
-----------------------------------------

pan3793 commented on PR #8622:
URL: https://github.com/apache/hadoop/pull/8622#issuecomment-5181319377

   Reviewed `FSOutputSummer.java` together with the `DataChecksum` factory and checksum sizes. The fix correctly turns the overflow into a fast `IllegalArgumentException`, and the test is well-targeted (`238609295 * 9 = 2147483655 > Integer.MAX_VALUE`). Nothing in the repo catches `NegativeArraySizeException`, so the exception-type change is safe. Findings below are simplifications, not correctness issues.
   
   **1. Dead `if (bufSize < 0)` / `if (checksumBufSize < 0)` branches (minor)**
   
   `Math.multiplyExact` throws `ArithmeticException` on overflow and never returns a wrapped negative, and `DataChecksum.newDataChecksum` already rejects `bytesPerChecksum <= 0` (`DataChecksum.java:112`), so `bufSize` is always positive after a successful multiply. Both negative checks are unreachable and can be dropped. (The PR description mentions a single `Preconditions.checkArgument`, but the implementation uses `Math.multiplyExact` + try/catch + the dead checks.)
   
   **2. Checksum-buffer try/catch is dead (nit)**
   
   `getChecksumSize()` returns `type.size`, which is `0` (NULL) or `4` (CRC32/CRC32C) only 

> NegativeArraySizeException thrown in FSOutputSummer.java given large file.bytes-per-checksum
> --------------------------------------------------------------------------------------------
>
>                 Key: HADOOP-18896
>                 URL: https://issues.apache.org/jira/browse/HADOOP-18896
>             Project: Hadoop Common
>          Issue Type: Bug
>    Affects Versions: 3.3.6
>            Reporter: rstest
>            Priority: Critical
>              Labels: pull-request-available
>
> Buffer size of FSOutputSummer equals to `file.bytes-per-checksum` times `BUFFER_NUM_CHUNKS`. A large `file.bytes-per-checksum` causes buffer size to overflow and crash with NegativeArraySizeException.
> To reproduce:
> 1. set `file.bytes-per-checksum` to 238609295
> 2. `mvn surefire:test -Dtest=org.apache.hadoop.hdfs.TestDecommissionWithStriped#testFileSmallerThanOneStripe`
> We created a PR that provides a fix which checks the buffer size is positive after multiplying `file.bytes-per-checksum` with `BUFFER_NUM_CHUNKS`



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org