You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Robert Chansler (JIRA)" <ji...@apache.org> on 2008/08/12 21:51:44 UTC
[jira] Created: (HADOOP-3938) Quotas for disk space management
Quotas for disk space management
--------------------------------
Key: HADOOP-3938
URL: https://issues.apache.org/jira/browse/HADOOP-3938
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Robert Chansler
Assignee: Raghu Angadi
Fix For: 0.19.0
Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631497#action_12631497 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
I don't see any global conf to disable quota restrictions. This is case before and after this patch. If image is wrong for some reason, then it won't start.. I think.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Status: Open (was: Patch Available)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631261#action_12631261 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
bq. "Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in violation of a quota (perhaps the fsimage was surreptitiously modified), the startup operation fails with an error report"
bq. So, how as an administrator could I ever fix such a problem?
my impression as well. I am not very sure why that was a requirement.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625811#action_12625811 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
bq. Somewhat more subtle than R's comment, it would not be possible to create a file in a directory with zero quota as creation requires that a block's-worth of quota is available. But one might think of setting a zero quota on a directory with only zero-length files.
Actually creating a file does not require block allocations. So space quota will be not be checked. It is possible to do {{dfs.create(); dfs.close()}}, even in a directory that reached its space quota. Is that what we want?
> Also, did you want to include the updated documentation in this patch?
sure, but not required in this jira.
Irrespective of whether it is possible or not, I think we need clarify/decide the policy : Should creating an empty file take up any space quota?
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625547#action_12625547 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
Patch for diskspace quotas is attached.
It works very similar to name space quotas implemented in HADOOP-3187 and [design document|https://issues.apache.org/jira/secure/attachment/12383512/QuotasDesign2.pdf] attached applies to this jira as well.
Description of "Space quotas" in the admin guide attached pretty much serves as the spec.
DFSClient is modified so that it throws the actual exception (including QuotaExceededException) when these exception occurs in other threads (most of the time).
TestQuota.java tests space quotas as well.
Currently there are differences in dfsadmin command names. will fix them.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3938) Quotas for disk space
management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631497#action_12631497 ]
rangadi edited comment on HADOOP-3938 at 9/16/08 11:33 AM:
----------------------------------------------------------------
I don't see any global conf to disable quota restrictions. This is the case before and after this patch. If the image is wrong for some reason, then it won't start.. it looks like.
was (Author: rangadi):
I don't see any global conf to disable quota restrictions. This is case before and after this patch. If image is wrong for some reason, then it won't start.. I think.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631804#action_12631804 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
We do have a test that uses a fixed NameNode image (used for testing upgrade from previous versions). It takes work to maintain as well. The quota violation can happen only with software bug and it is not fatal, I don't think it is a must to have a unit test for it in this jira. I will manually test of course.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Thanks Konstantin. Updated patch modifies {{setQuota()}} as Konstantin suggested. It looks better now. setQuota() does verifies only the the quota that is modified. So if space quota is not modified then it will not be checked. This is essential to deal with problems in quota management (previous run of HDFS).
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625819#action_12625819 ]
Robert Chansler commented on HADOOP-3938:
-----------------------------------------
Sorry, I confused the issue.
Create should not require any space quota. The quota should be checked only when a new block is requested.
I'll attach a new version of the guide with the clarification!
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631500#action_12631500 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
> I'm going to jump on the pile and ask that a test plan be included for this feature.
It is the same as in HADOOP-3187. The unit test covers pretty much all of them. Does it sound good enough? Only the scale test is not done.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Status: Open (was: Patch Available)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632851#action_12632851 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
+1 this looks good now.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Resolution: Fixed
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Resolved (was: Patch Available)
I just committed this.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3938:
------------------------------------
Attachment: hdfs_quota_admin_guide.xml
Draft user guide.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630438#action_12630438 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
- In {{FSDirectory.replaceNode()}} you calculate {{diskspaceConsumed()}} even if you do not need to {{updateDiskspace}}.
+1 the rest looks good to me.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Nigel Daley (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631494#action_12631494 ]
Nigel Daley commented on HADOOP-3938:
-------------------------------------
I'm going to jump on the pile and ask that a test plan be included for this feature.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632890#action_12632890 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
How to test with quota violations in fsimage :
===================================
* Comment out two places where {{QuotaExceededException}} is thrown in {{INodeDirectoryWithQuota}}
* Run a fresh HDFS and create a few files and directories such that it exceeds quotas for a few directories. For e.g. : {noformat}
bin/hadoop fs -mkdir quota3-3k
bin/hadoop dfsadmin -setQuota 3 quota3-3k
bin/hadoop fs -mkdir quota3-3k/emptyDir
bin/hadoop fs -put /dev/null quota3-3k/emptyFile
bin/hadoop dfsadmin -setSpaceQuota 3072 quota3-3k
bin/hadoop fs -put ~/cws/tmp/5Mb quota3-3k/
bin/hadoop fs -mkdir quota3-3k/quota2
bin/hadoop dfsadmin -setQuota 2 quota3-3k/quota2
bin/hadoop fs -mkdir quota3-3k/quota2/emptyDir
bin/hadoop fs -put /dev/null quota3-3k/quota2/emptyFile
bin/hadoop fs -mkdir quota3-3k/quota-5k
bin/hadoop dfsadmin -setSpaceQuota 5120 quota3-3k/quota-5k
bin/hadoop fs -put ~/cws/tmp/5Mb quota3-3k/quota-5k
bin/hadoop fs -mkdir quota3-3k/quota-5k/emptyDir
{noformat}
* Stop the cluster and start it again.
* The cluster should restart fine. You should see some warnings for {{quota3-3k}} etc in NameNode log.
* Adding any more files or data to such directories should fail.
* You should be able to either increase the quotas or delete some files to satisfy the requirements.
* Once these are fixed, there should not be any more warnings when NameNode restarts.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
Status: Patch Available (was: Open)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Updated patch resolves a conflict with a recent commit.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631875#action_12631875 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
Steve, what I meant is that you cannot (naturally, using hdfs methods) create an image which violates a quota.
You can obviously write an arbitrary number instead of the existing quota value directly in the fsimage file if you know where, but this is not a "valid" way.
BTW, this issue is about hdfs (distributed) space quotas, not sure that determining "disk space use" helps a lot.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625808#action_12625808 ]
Robert Chansler commented on HADOOP-3938:
-----------------------------------------
Somewhat more subtle than R's comment, it would not be possible to create a file in a directory with zero quota as creation requires that a block's-worth of quota is available. But one might think of setting a zero quota on a directory with only zero-length files. In general, new file creations can be prevented by setting a directory's byte quota to the sum of existing file lengths, or the name quota to the number of existing files and directories.
Also, did you want to include the updated documentation in this patch?
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631220#action_12631220 ]
Pete Wyckoff commented on HADOOP-3938:
--------------------------------------
>From the user spec:
"Quotas are persistent with the fsimage. When starting, if the fsimage is immediately in violation of a quota (perhaps the fsimage was surreptitiously modified), the startup operation fails with an error report"
So, how as an administrator could I ever fix such a problem? Are there administration commands that are going to work directly on the fsimage without a namenode? This seems like a very poor requirement.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631487#action_12631487 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
Imo, you should be able to start name-node with quotas turned off (in config) and then be able to correct quotas for the faulty directory.
Does the patch allow that?
We should also check namespace quotas for the same problem.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12633335#action_12633335 ]
Hudson commented on HADOOP-3938:
--------------------------------
Integrated in Hadoop-trunk #611 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/611/])
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Status: Patch Available (was: Open)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3938:
------------------------------------
Attachment: (was: hdfs_quota_admin_guide.xml)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629627#action_12629627 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
Thanks for the review Konstantin.
Regd (8), (9) etc : Could you suggest a value for quota that implies 'do not modify quota'.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Steve Loughran (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631715#action_12631715 ]
Steve Loughran commented on HADOOP-3938:
----------------------------------------
Konstantin said
> A unit test would be hard to write for the case since there are no valid ways to reproduce the condition.
you could a functional test with a vmware/xen image, though it would take a lot of work. The alternate tactic is to have a mock implementation of the code to determine disk space use, and simulate failures when a node comes up.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3938:
------------------------------------
Attachment: hdfs_quota_admin_guide.pdf
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3938:
------------------------------------
Release Note: Introducted byte space quotas for directories. The count shell command modified to report both name and byte quotas. (was: Implements byte space quotas for directories. The count shell command modified to report both name and byte quotas.)
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed, Incompatible change])
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625548#action_12625548 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
User guide says : "A quota of zero forces the directory tree to remain empty of files."
A directory with a space quota of '0' bytes can have zero length files (though setting the quota to 0 is not actually allowed).
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632493#action_12632493 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
Updated patch has following changes over the previous patch :
# Quota violations are tolerated while loading filesystem during start up.
# {{updateCountForINodeWithQuota()}} called during initialization is rewritten.
# {{updateCountForINodeWithQuota()}} is now called after loading edits, rather than right after loading fsimage.
# Added one more test case to handle the case when quotas specified are too large.
# user_guide is slightly updated.
The handling of quota violations are tested by running a modified NameNode that does not generate a {{QuotaExceededException}}. will have another comment on this.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631491#action_12631491 ]
Pete Wyckoff commented on HADOOP-3938:
--------------------------------------
That's weird to set quotas with the configuration turned off? I hope this is implemented as Konstantin said, but some unit tests for this condition would probably help future implementors :)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Attached patch does the following :
- includes updated hdfs_quota_admin_guide.xml from Rob
- Konstantin's review : invoke {{diskspaceConsumed()}} only when required (it is not required only in some error cases).
- fix findbugs warning. {{ant test-patch}} is also run.
Regd Nicholas' comment : Using {{Long}} object also works. Not sure if there is any real difference.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631232#action_12631232 ]
Hadoop QA commented on HADOOP-3938:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12390029/HADOOP-3938.patch
against trunk revision 695690.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 22 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3265/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3265/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3265/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3265/console
This message is automatically generated.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Patch updated for trunk.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Hadoop Flags: [Incompatible change, Reviewed] (was: [Incompatible change])
Status: Patch Available (was: Open)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler updated HADOOP-3938:
------------------------------------
Attachment: hdfs_quota_admin_guide.xml
Corrects a grammar mistake and clarifies that file creation does not require space quota.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632494#action_12632494 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
Konstantin, could you take another look at the patch? thanks.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631509#action_12631509 ]
Robert Chansler commented on HADOOP-3938:
-----------------------------------------
I reviewed the patch, and the offending words about not starting are not in the new documentation. The new documentation as written only requires that the quota violation be logged.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630533#action_12630533 ]
Hadoop QA commented on HADOOP-3938:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12389870/HADOOP-3938.patch
against trunk revision 694562.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 22 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 2 new Findbugs warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3257/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3257/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3257/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch/3257/console
This message is automatically generated.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Patch updated : minor change for {{INodeDirectoryWithQuota.setQuota()}}. It verifies only if the new quota is more restrictive.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12625812#action_12625812 ]
dhruba borthakur commented on HADOOP-3938:
------------------------------------------
My vote is that creating a empty file does not take any disk-quota. (However, it does take up a namespace quota).
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3938) Quotas for disk space
management
Posted by "Pete Wyckoff (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631491#action_12631491 ]
wyckoff edited comment on HADOOP-3938 at 9/16/08 11:23 AM:
----------------------------------------------------------------
That's weird to set quotas with the configuration turned off? I hope this is implemented as Konstantin said, but some unit tests for this use case would probably help future implementors :)
was (Author: wyckoff):
That's weird to set quotas with the configuration turned off? I hope this is implemented as Konstantin said, but some unit tests for this condition would probably help future implementors :)
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12630410#action_12630410 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-3938:
------------------------------------------------
> 8 , 9 : We use Long.MIN_VALUE to indicate "don't modify". This reduces quite a bit of similar looking code.
You could use the wrapper class Long and use null for "don't modify". i.e.
{code}
private void unprotectedSetQuota(String src, Long nsQuota, Long dsQuota)
{code}
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629335#action_12629335 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
- It is better to implement
{code}
public ContentSummary(long length, long fileCount, long directoryCount) {
this(length, fileCount, directoryCount, length, -1)
}
{code}
- DFSClient in {{locateFollowingBlock()}} you do not need to throw unwrapped exception in order to check whether it is a RemoteException.
And also you do not throw if the remote exception is not of any of the 4 types considered.
{code}
try {
return namenode.addBlock(src, clientName);
} catch (RemoteException re) {
IOException ue = re.unwrapRemoteException(FileNotFoundException.class,
AccessControlException.class,
QuotaExceededException.class);
if(re != ue)
throw ue;
ue = re.unwrapRemoteException(NotReplicatedYetException.class);
if(--retries == 0 && re == ue) // not a NotReplicatedYetException
throw re;
LOG.warn("NotReplicatedYetException sleeping " + src + " retries left " + retries);
try {
Thread.sleep(sleeptime);
sleeptime *= 2;
} catch (InterruptedException ie) {}
}
{code}
- JavaDoc for {{QuotaExceededException}} should reflect new space quota related semantics.
- In {{QuotaExceededException.getMessage()}} {{" file count="}} instead of {{" namespace count="}} may sound better.
- FSDirectory does not need to {{import StringUtils}}.
- {{FSDirectory.unprotectedAddFile()}} should probably not add blocks first and then remove them
if the space quota is violated. The blocks can be added once the node is successfully created,
you may need to pass the size of the file to {{addNode()}}.
- {{numItemsInTree()}} followed by {{diskspaceInTree()}} is called on several occasions: {{INodeDirectoryWithQuota(), addChild(), removeChild()}}.
This is very inefficient, tree traversal should be be done only once.
You can use something like the {{TwoCounters}} class but with a more meaningful name, say DirectoryQuota.
I think {{numItemsInTree()}} and {{diskspaceInTree()}} should be merged in one method preferably non-recursive.
- unprotectedSetQuota() should take 2 longs rather than a long and a boolean
{code}
private void unprotectedSetQuota(String src, long nsQuota, long dsQuota)
{code}
is in other places, e.g. addToParent().
- I am not sure we need to add an extra pair of set/clear-DiskspaceQuota(), may be we just need
an extra parameter in the old {{setQuota(src, nsQuota, dsQuota)}}
- Should we check space quota when we start an append?
- NameNode redundantly imports BlockCommand (not introduced by this patch).
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Issue Comment Edited: (HADOOP-3938) Quotas for disk space
management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12629335#action_12629335 ]
shv edited comment on HADOOP-3938 at 9/8/08 5:23 PM:
---------------------------------------------------------------------
# It is better to implement
{code}
public ContentSummary(long length, long fileCount, long directoryCount) {
this(length, fileCount, directoryCount, length, -1)
}
{code}
# DFSClient in {{locateFollowingBlock()}} you do not need to throw unwrapped exception in order to check whether it is a RemoteException.
And also you do not throw if the remote exception is not of any of the 4 types considered.
{code}
try {
return namenode.addBlock(src, clientName);
} catch (RemoteException re) {
IOException ue = re.unwrapRemoteException(FileNotFoundException.class,
AccessControlException.class,
QuotaExceededException.class);
if(re != ue)
throw ue;
ue = re.unwrapRemoteException(NotReplicatedYetException.class);
if(--retries == 0 && re == ue) // not a NotReplicatedYetException
throw re;
LOG.warn("NotReplicatedYetException sleeping " + src + " retries left " + retries);
try {
Thread.sleep(sleeptime);
sleeptime *= 2;
} catch (InterruptedException ie) {}
}
{code}
# JavaDoc for {{QuotaExceededException}} should reflect new space quota related semantics.
# In {{QuotaExceededException.getMessage()}} {{" file count="}} instead of {{" namespace count="}} may sound better.
# FSDirectory does not need to {{import StringUtils}}.
# {{FSDirectory.unprotectedAddFile()}} should probably not add blocks first and then remove them
if the space quota is violated. The blocks can be added once the node is successfully created,
you may need to pass the size of the file to {{addNode()}}.
# {{numItemsInTree()}} followed by {{diskspaceInTree()}} is called on several occasions: {{INodeDirectoryWithQuota(), addChild(), removeChild()}}.
This is very inefficient, tree traversal should be be done only once.
You can use something like the {{TwoCounters}} class but with a more meaningful name, say DirectoryQuota.
I think {{numItemsInTree()}} and {{diskspaceInTree()}} should be merged in one method preferably non-recursive.
# unprotectedSetQuota() should take 2 longs rather than a long and a boolean
{code}
private void unprotectedSetQuota(String src, long nsQuota, long dsQuota)
{code}
as in other places, e.g. addToParent().
# I am not sure we need to add an extra pair of set/clear-DiskspaceQuota(), may be we just need
an extra parameter in the old {{setQuota(src, nsQuota, dsQuota)}}
# Should we check space quota when we start an append?
# NameNode redundantly imports BlockCommand (not introduced by this patch).
was (Author: shv):
- It is better to implement
{code}
public ContentSummary(long length, long fileCount, long directoryCount) {
this(length, fileCount, directoryCount, length, -1)
}
{code}
- DFSClient in {{locateFollowingBlock()}} you do not need to throw unwrapped exception in order to check whether it is a RemoteException.
And also you do not throw if the remote exception is not of any of the 4 types considered.
{code}
try {
return namenode.addBlock(src, clientName);
} catch (RemoteException re) {
IOException ue = re.unwrapRemoteException(FileNotFoundException.class,
AccessControlException.class,
QuotaExceededException.class);
if(re != ue)
throw ue;
ue = re.unwrapRemoteException(NotReplicatedYetException.class);
if(--retries == 0 && re == ue) // not a NotReplicatedYetException
throw re;
LOG.warn("NotReplicatedYetException sleeping " + src + " retries left " + retries);
try {
Thread.sleep(sleeptime);
sleeptime *= 2;
} catch (InterruptedException ie) {}
}
{code}
- JavaDoc for {{QuotaExceededException}} should reflect new space quota related semantics.
- In {{QuotaExceededException.getMessage()}} {{" file count="}} instead of {{" namespace count="}} may sound better.
- FSDirectory does not need to {{import StringUtils}}.
- {{FSDirectory.unprotectedAddFile()}} should probably not add blocks first and then remove them
if the space quota is violated. The blocks can be added once the node is successfully created,
you may need to pass the size of the file to {{addNode()}}.
- {{numItemsInTree()}} followed by {{diskspaceInTree()}} is called on several occasions: {{INodeDirectoryWithQuota(), addChild(), removeChild()}}.
This is very inefficient, tree traversal should be be done only once.
You can use something like the {{TwoCounters}} class but with a more meaningful name, say DirectoryQuota.
I think {{numItemsInTree()}} and {{diskspaceInTree()}} should be merged in one method preferably non-recursive.
- unprotectedSetQuota() should take 2 longs rather than a long and a boolean
{code}
private void unprotectedSetQuota(String src, long nsQuota, long dsQuota)
{code}
is in other places, e.g. addToParent().
- I am not sure we need to add an extra pair of set/clear-DiskspaceQuota(), may be we just need
an extra parameter in the old {{setQuota(src, nsQuota, dsQuota)}}
- Should we check space quota when we start an append?
- NameNode redundantly imports BlockCommand (not introduced by this patch).
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12632769#action_12632769 ]
Raghu Angadi commented on HADOOP-3938:
--------------------------------------
tes-patch output from my machine :
{noformat}
[exec] +1 overall.
[exec] +1 @author. The patch does not contain any @author tags.
[exec] +1 tests included. The patch appears to include 22 new or modified tests.
[exec] +1 javadoc. The javadoc tool did not generate any warning messages.
[exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings.
[exec] +1 findbugs. The patch does not introduce any new Findbugs warnings.
{noformat}
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631502#action_12631502 ]
Konstantin Shvachko commented on HADOOP-3938:
---------------------------------------------
On second thought my proposal is not good since it is exactly the way to produce such incorrect state.
Instead, we should just remove the restriction and let the server start with a warning.
A unit test would be hard to write for the case since there are no valid ways to reproduce the condition.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-3938) Quotas for disk space management
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12631503#action_12631503 ]
Robert Chansler commented on HADOOP-3938:
-----------------------------------------
I'm guilty of having caused the confusion.
In any case, the consensus seems to be for policy to be the same for both space and name quotas, and that the proper policy is to log any violations that are observed at start up, but to resume normal operation.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-3938) Quotas for disk space management
Posted by "Raghu Angadi (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Raghu Angadi updated HADOOP-3938:
---------------------------------
Attachment: HADOOP-3938.patch
Updated patch incorporates detailed review from Konstantin.
- 1, 3, 4, 5, 6, 11 : done
- 2: done. actual changes are slightly different to keep the lines changed to minimum
- 7 : done. It is still recursive. I am not sure if non-recursive implementation buys anything. We can avoid recursive calls to INodeFile if we want to.
- 8 , 9 : We use {{Long.MIN_VALUE}} to indicate "don't modify". This reduces quite a bit of similar looking code.
- 10 : It is checked eventually in side {{startFileInternal()}} if the file being appended has a partially filled block.
> Quotas for disk space management
> --------------------------------
>
> Key: HADOOP-3938
> URL: https://issues.apache.org/jira/browse/HADOOP-3938
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Robert Chansler
> Assignee: Raghu Angadi
> Fix For: 0.19.0
>
> Attachments: HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, HADOOP-3938.patch, hdfs_quota_admin_guide.pdf, hdfs_quota_admin_guide.xml
>
>
> Directory quotas for bytes limit the number of bytes used by files in and below the directory. Operation is independent of name quotas (HADOOP-3187), but the implementation is parallel. Each file is charged according to its length multiplied by its intended replication factor.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.