You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ASF GitHub Bot (Jira)" <ji...@apache.org> on 2023/09/14 00:46:00 UTC
[jira] [Commented] (HADOOP-18896) NegativeArraySizeException thrown in FSOutputSummer.java given large file.bytes-per-checksum
[ https://issues.apache.org/jira/browse/HADOOP-18896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17764925#comment-17764925 ]
ASF GitHub Bot commented on HADOOP-18896:
-----------------------------------------
teamconfx opened a new pull request, #6064:
URL: https://github.com/apache/hadoop/pull/6064
### Description of PR
Buffer size of FSOutputSummer equals to `file.bytes-per-checksum` times `BUFFER_NUM_CHUNKS`. A large `file.bytes-per-checksum` causes buffer size to overflow and crash with NegativeArraySizeException.
To reproduce:
1. set `file.bytes-per-checksum` to 238609295
2. run `mvn surefire:test -Dtest=org.apache.hadoop.hdfs.TestDecommissionWithStriped#testFileSmallerThanOneStripe`
This PR provides a fix which checks the buffer size is positive after multiplying `file.bytes-per-checksum` with `BUFFER_NUM_CHUNKS`
### How was this patch tested?
Unit test
### For code changes:
- [x] Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
- [ ] Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
- [ ] If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion under ASF 2.0?
- [ ] If applicable, have you updated the LICENSE, LICENSE-binary, NOTICE-binary files?
> NegativeArraySizeException thrown in FSOutputSummer.java given large file.bytes-per-checksum
> --------------------------------------------------------------------------------------------
>
> Key: HADOOP-18896
> URL: https://issues.apache.org/jira/browse/HADOOP-18896
> Project: Hadoop Common
> Issue Type: Bug
> Affects Versions: 3.3.6
> Reporter: ConfX
> Priority: Critical
>
> Buffer size of FSOutputSummer equals to `file.bytes-per-checksum` times `BUFFER_NUM_CHUNKS`. A large `file.bytes-per-checksum` causes buffer size to overflow and crash with NegativeArraySizeException.
> To reproduce:
> 1. set `file.bytes-per-checksum` to 238609295
> 2. `mvn surefire:test -Dtest=org.apache.hadoop.hdfs.TestDecommissionWithStriped#testFileSmallerThanOneStripe`
> We linked this issue to a PR that provides a fix which checks the buffer size is positive after multiplying `file.bytes-per-checksum` with `BUFFER_NUM_CHUNKS`
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org