You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-issues@hadoop.apache.org by "Harshit Gupta (Jira)" <ji...@apache.org> on 2023/02/21 10:53:00 UTC
[jira] [Commented] (HADOOP-18637) S3A to support upload of files greater than 2 GB using DiskBlocks
[ https://issues.apache.org/jira/browse/HADOOP-18637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17691517#comment-17691517 ]
Harshit Gupta commented on HADOOP-18637:
----------------------------------------
S3ABlockOutputStream creates three different types of DataBlock depending upon the {{fs.s3a.fast.upload.buffer}} which defaults to disk, we can create an empty file for the same size and limit the Buffer size to {{Integer.MAX_VALUE}} . *For other buffer types should we deny uploads larger than 2 Gigs or should we add the support there as well?* like for {{ByteArrayBlock}} which writes directly to the {{S3AByteArrayOutputStream}} which will be again initialized with {{Integer.MAX_Value}} .The same goes for {{ByteBufferBlock}} as well. One thing to make sure of here is that it's never gonna write something larger than {{Integer.MAX_VALUE}} as the calling function to write has the signature {{public synchronized void write(byte[] source, int offset, int len)}} (S3ABlockOutputStream).
*This is just for compatibility with non-AWS s3 stores.*
> S3A to support upload of files greater than 2 GB using DiskBlocks
> -----------------------------------------------------------------
>
> Key: HADOOP-18637
> URL: https://issues.apache.org/jira/browse/HADOOP-18637
> Project: Hadoop Common
> Issue Type: Improvement
> Components: fs/s3
> Reporter: Harshit Gupta
> Assignee: Harshit Gupta
> Priority: Major
>
> Use S3A Diskblocks to support the upload of files greater than 2 GB using DiskBlocks. Currently, the max upload size of a single block is ~2GB.
> cc: [~mthakur] [~stevel@apache.org] [~mehakmeet]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org