You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-issues@hadoop.apache.org by "Steve Loughran (JIRA)" <ji...@apache.org> on 2016/11/29 11:56:58 UTC

[jira] [Commented] (HADOOP-13261) save partition split size on multipart uploads

    [ https://issues.apache.org/jira/browse/HADOOP-13261?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15705095#comment-15705095 ] 

Steve Loughran commented on HADOOP-13261:
-----------------------------------------

it'd be expensive to query though, as you do not get this information back on a LIST; you'd need to do a HEAD on the file —way too expensive to use in split calculation. 

Where it could be used is in copy and perhaps an s3-specific distcp, where the partition size could be propagated

> save partition split size on multipart uploads
> ----------------------------------------------
>
>                 Key: HADOOP-13261
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13261
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs/s3
>    Affects Versions: 2.8.0
>            Reporter: Steve Loughran
>            Priority: Minor
>
> On multipart uploads, save the split size as a metadata value. This would allow split calculation optimized for the partitions to be performed in some bulk operation.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org