[jira] [Created] (HADOOP-17937) .ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct buffer memory

["Steve Loughran (Jira)" <ji...@apache.org>]

Steve Loughran created HADOOP-17937:
---------------------------------------

             Summary: .ITestS3ADeleteFilesOneByOne. testBulkRenameAndDelete OOM: Direct buffer memory
                 Key: HADOOP-17937
                 URL: https://issues.apache.org/jira/browse/HADOOP-17937
             Project: Hadoop Common
          Issue Type: Bug
          Components: fs/s3, test
         Environment: fs.s3a.fast.upload.buffer = "bytebuffer"
            Reporter: Steve Loughran


on a test setup with bytebuffer, the parallel zero-byte file create phase OOMed

fs.s3a.fast.upload.buffer = "bytebuffer" [core-site.xml]
fs.s3a.fast.upload.active.blocks = "8" [core-site.xml]
fs.s3a.multipart.size = "32M" [core-site.xml]

Root cause: bytebuffer is being allocated on block creation, so every empty file took up 32MB of off-heap storage only for this to be released unused in close()

If this allocation was postponed until the first write(), then empty files wouldn't need any memory allocation. Do the same on-demand creation for byte arrays and filesystem would also have benefits.

this has implications for  HADOOP-17195, which has abfs using a fork of the buffering code 

changing the code there to be on-demand would be a good incentive for s3a to adopt



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org