You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "ConfX (Jira)" <ji...@apache.org> on 2023/07/18 15:37:00 UTC
[jira] [Updated] (HADOOP-18804) Out of Memory when mistakenly set io.file.buffer.size to a large number
[ https://issues.apache.org/jira/browse/HADOOP-18804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
ConfX updated HADOOP-18804:
---------------------------
Attachment: reproduce.sh
> Out of Memory when mistakenly set io.file.buffer.size to a large number
> -----------------------------------------------------------------------
>
> Key: HADOOP-18804
> URL: https://issues.apache.org/jira/browse/HADOOP-18804
> Project: Hadoop Common
> Issue Type: Bug
> Components: common, fs
> Reporter: ConfX
> Priority: Critical
> Attachments: reproduce.sh
>
>
> h2. What happened:
> When setting io.file.buffer.size to a large number, BufferedIOStatisticsOutputStream in Hcommon throws an out-of-memory exception due to inappropriate checking and handling.
> The config is used to initialize a file system by passing it as one of the parameters bufferSize.
> h2. Buggy code:
> In RawLocalFileSystem.java
> {code:java}
> private FSDataOutputStream create(Path f, boolean overwrite,
> boolean createParent, int bufferSize, short replication, long blockSize,
> Progressable progress, FsPermission permission) throws IOException {
> ...
> return new FSDataOutputStream(new BufferedIOStatisticsOutputStream(
> createOutputStreamWithMode(f, false, permission), bufferSize, true), <<--- creates a BufferedIOStatisticsOutputStream with bufferSize, often set to config io.file.buffer.size
> statistics);
> } {code}
> In BufferedIOStatisticsOutputStream.java:
> {code:java}
> public class BufferedIOStatisticsOutputStream extends BufferedOutputStream
> implements IOStatisticsSource, Syncable, StreamCapabilities {
> ...
> public BufferedIOStatisticsOutputStream(
> final OutputStream out,
> final int size,
> final boolean downgradeSyncable) {
> super(out, size); <<--- init the BufferedOutputStream with a huge buffer size
> ...
> }{code}
> h2. StackTrace:
> {code:java}
> java.lang.OutOfMemoryError: Java heap space
> at java.base/java.io.BufferedOutputStream.<init>(BufferedOutputStream.java:75)
> at org.apache.hadoop.fs.statistics.BufferedIOStatisticsOutputStream.<init>(BufferedIOStatisticsOutputSt
> ream.java:78)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:428)
> at org.apache.hadoop.fs.RawLocalFileSystem.create(RawLocalFileSystem.java:413)
> at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:1175)
> at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:183)
> at org.apache.hadoop.fs.contract.ContractTestUtils.writeDataset(ContractTestUtils.java:152)
> at org.apache.hadoop.fs.contract.AbstractContractRenameTest.expectRenameUnderFileFails(AbstractContract
> RenameTest.java:335)
> ...{code}
> h2. Reproduce:
> (1) Set io.file.buffer.size to a large value, e.g., 2112001717
> (2) Run a simple test that exercises this parameter, e.g. org.apache.hadoop.fs.contract.rawlocal.TestRawlocalContractRename#testRenameFileUnderFile
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org