You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/10/22 13:53:47 UTC
[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1516: HDDS-3731. [doc]add storage space quota document.
captainzmc opened a new pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516
## What changes were proposed in this pull request?
The space Quota feature is now nearly complete, so add a description of how to use it
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3731
## How was this patch tested?
no need test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510568534
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
Review comment:
The name space Quota feature is currently not available and is under development. We can add this after it is complete.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510364763
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
+
+Administrators should be able to define how much storage space a Volume or Bucket can use.
+
+## Client usage
+### Storage Space level quota
+Storage space level quotas allow the use of units such as KB (k), MB (m), GB (g), TB (t), PB (p), etc. Represents how many storage Spaces will be used.
+
+#### Volume Storage Space level quota
+```shell
+bin/ozone sh volume create --space-quota 5m /volume1
+```
+This means setting the storage space of Volume1 to 5MB
+
+```shell
+bin/ozone sh volume setquota --space-quota 10g /volume1
+```
+This behavior changes the quota of Volume1 to 10GB.
+
+#### Bucket Storage Space level quota
+```shell
+bin/ozone sh bucket create --space-quota 5m /volume1/bucket1
+```
+That means bucket1 allows us to use 5MB of storage.
+
+```shell
+bin/ozone sh bucket setquota --space-quota 10g /volume1/bucket1
+```
+This behavior changes the quota for Bucket1 to 10GB
+
+A bucket quota should not be greater than its Volume quota. Let's look at an example. If we have a 10MB Volume and create five buckets under that Volume with a quota of 5MB, the total quota is 25MB. In this case, the bucket creation will always succeed, and we check the quota for bucket and volume when the data is actually written. Each write needs to check whether the current bucket is exceeding the limit and the current total volume usage is exceeding the limit.
+
+#### Clear the quota for Volume1. The Bucket cleanup command is similar.
+```shell
+bin/ozone sh volume clrquota --space-quota /volume1
+```
Review comment:
Can you add a section on how to check the quota and usage?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao merged pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao merged pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510363980
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
+
+Administrators should be able to define how much storage space a Volume or Bucket can use.
+
+## Client usage
+### Storage Space level quota
+Storage space level quotas allow the use of units such as KB (k), MB (m), GB (g), TB (t), PB (p), etc. Represents how many storage Spaces will be used.
+
+#### Volume Storage Space level quota
+```shell
+bin/ozone sh volume create --space-quota 5m /volume1
+```
+This means setting the storage space of Volume1 to 5MB
+
+```shell
+bin/ozone sh volume setquota --space-quota 10g /volume1
+```
+This behavior changes the quota of Volume1 to 10GB.
+
+#### Bucket Storage Space level quota
+```shell
+bin/ozone sh bucket create --space-quota 5m /volume1/bucket1
+```
+That means bucket1 allows us to use 5MB of storage.
+
+```shell
+bin/ozone sh bucket setquota --space-quota 10g /volume1/bucket1
+```
+This behavior changes the quota for Bucket1 to 10GB
+
+A bucket quota should not be greater than its Volume quota. Let's look at an example. If we have a 10MB Volume and create five buckets under that Volume with a quota of 5MB, the total quota is 25MB. In this case, the bucket creation will always succeed, and we check the quota for bucket and volume when the data is actually written. Each write needs to check whether the current bucket is exceeding the limit and the current total volume usage is exceeding the limit.
Review comment:
what if the case of volume quota = 10MB but create a bucket of quota 20MB. Will the bucket create fail?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#issuecomment-714884328
Thanks @xiaoyuyao 's review. I have submitted a new commit fix the above issues.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510361614
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
Review comment:
Do we have a section for name space quota?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#issuecomment-717361668
Thanks @captainzmc for the update. LGTM, +1.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] xiaoyuyao commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
xiaoyuyao commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510360994
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
+
+Administrators should be able to define how much storage space a Volume or Bucket can use.
+
+## Client usage
+### Storage Space level quota
+Storage space level quotas allow the use of units such as KB (k), MB (m), GB (g), TB (t), PB (p), etc. Represents how many storage Spaces will be used.
Review comment:
how many => how much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510568737
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
+
+Administrators should be able to define how much storage space a Volume or Bucket can use.
+
+## Client usage
+### Storage Space level quota
+Storage space level quotas allow the use of units such as KB (k), MB (m), GB (g), TB (t), PB (p), etc. Represents how many storage Spaces will be used.
+
+#### Volume Storage Space level quota
+```shell
+bin/ozone sh volume create --space-quota 5m /volume1
+```
+This means setting the storage space of Volume1 to 5MB
+
+```shell
+bin/ozone sh volume setquota --space-quota 10g /volume1
+```
+This behavior changes the quota of Volume1 to 10GB.
+
+#### Bucket Storage Space level quota
+```shell
+bin/ozone sh bucket create --space-quota 5m /volume1/bucket1
+```
+That means bucket1 allows us to use 5MB of storage.
+
+```shell
+bin/ozone sh bucket setquota --space-quota 10g /volume1/bucket1
+```
+This behavior changes the quota for Bucket1 to 10GB
+
+A bucket quota should not be greater than its Volume quota. Let's look at an example. If we have a 10MB Volume and create five buckets under that Volume with a quota of 5MB, the total quota is 25MB. In this case, the bucket creation will always succeed, and we check the quota for bucket and volume when the data is actually written. Each write needs to check whether the current bucket is exceeding the limit and the current total volume usage is exceeding the limit.
Review comment:
Yes, the quota value of bucket cannot be greater than the quota value of volume
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1516: HDDS-3731. [doc]add storage space quota document.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1516:
URL: https://github.com/apache/hadoop-ozone/pull/1516#discussion_r510568534
##########
File path: hadoop-hdds/docs/content/feature/Quota.md
##########
@@ -0,0 +1,67 @@
+---
+title: "Quota in Ozone"
+date: "2020-October-22"
+weight: 4
+summary: Quota in Ozone
+icon: user
+menu:
+ main:
+ parent: Features
+summary: Introduction to Ozone Quota
+---
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one or more
+ contributor license agreements. See the NOTICE file distributed with
+ this work for additional information regarding copyright ownership.
+ The ASF licenses this file to You under the Apache License, Version 2.0
+ (the "License"); you may not use this file except in compliance with
+ the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing, software
+ distributed under the License is distributed on an "AS IS" BASIS,
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ See the License for the specific language governing permissions and
+ limitations under the License.
+-->
+
+So far, we know that Ozone allows users to create volumes, buckets, and keys. A Volume usually contains several buckets, and each Bucket also contains a certain number of keys. Obviously, it should allow the user to define quotas (for example, how many buckets can be created under a Volume or how much space can be used by a Bucket), which is a common requirement for storage systems.
+
+## Currently supported
+1. Storage Space level quota
Review comment:
The name space Quota feature is currently not available and is under development. We can refine the document after it is complete.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org