You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Todd Lipcon (JIRA)" <ji...@apache.org> on 2013/04/09 23:50:17 UTC
[jira] [Commented] (HDFS-4305) Add a configurable limit on number
of blocks per file, and min block size
[ https://issues.apache.org/jira/browse/HDFS-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13627159#comment-13627159 ]
Todd Lipcon commented on HDFS-4305:
-----------------------------------
This looks good to me.
One possible change: we can set the default min block size to 1MB or so, and then add a minimum of 0 to the hdfs-site.xml in src/test/resources to avoid having to rewrite a bunch of tests.
It would be an incompatible change, but perhaps a good one. What do folks think?
> Add a configurable limit on number of blocks per file, and min block size
> -------------------------------------------------------------------------
>
> Key: HDFS-4305
> URL: https://issues.apache.org/jira/browse/HDFS-4305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
> Affects Versions: 1.0.4, 3.0.0, 2.0.2-alpha
> Reporter: Todd Lipcon
> Assignee: Andrew Wang
> Priority: Minor
> Attachments: hdfs-4305-1.patch
>
>
> We recently had an issue where a user set the block size very very low and managed to create a single file with hundreds of thousands of blocks. This caused problems with the edit log since the OP_ADD op was so large (HDFS-4304). I imagine it could also cause efficiency issues in the NN. To prevent users from making such mistakes, we should:
> - introduce a configurable minimum block size, below which requests are rejected
> - introduce a configurable maximum number of blocks per file, above which requests to add another block are rejected (with a suitably high default as to not prevent legitimate large files)
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira