You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/06/09 16:27:49 UTC
[GitHub] [hadoop-ozone] Simon0806 opened a new pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Simon0806 opened a new pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041
## What changes were proposed in this pull request?
Add quota option to ozone sh volume command.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-3725
## How was this patch tested?
unit tests,
manual tests
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#issuecomment-651672572
Adding a few paragraphs to the jira (or to the PR) would help to understand the goal. It seems to introduce new type of quota (count based instead of byte based) which is not clear from the description
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r451330394
##########
File path: hadoop-hdds/docs/content/shell/VolumeCommands.md
##########
@@ -42,7 +42,7 @@ assign it to a user.
| Uri | The name of the volume. |
{{< highlight bash >}}
-ozone sh volume create --quota=1TB --user=bilbo /hive
+ozone sh volume create -ssq=1TB --user=bilbo /hive
Review comment:
in the previous design, quota just describes the Space Quota, if we want to describe the Count Quota, which one is better to use ? quota is a little ambiguous to describe which one it wants to express.
IMO, I prefer to use spaceQuota and countQuota instead
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r441408302
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##########
@@ -255,10 +255,9 @@ public void testSetVolumeQuota()
throws IOException {
String volumeName = UUID.randomUUID().toString();
store.createVolume(volumeName);
- store.getVolume(volumeName).setQuota(
- OzoneQuota.parseQuota("100000000 BYTES"));
+ store.getVolume(volumeName).setQuota(OzoneQuota.parseQuota("1GB", 0L));
OzoneVolume volume = store.getVolume(volumeName);
- Assert.assertEquals(100000000L, volume.getQuota());
+ Assert.assertEquals(1073741824L, volume.getStoragespaceQuota());
Review comment:
Accept.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r437873358
##########
File path: hadoop-hdds/docs/content/shell/VolumeCommands.md
##########
@@ -42,7 +42,7 @@ assign it to a user.
| Uri | The name of the volume. |
{{< highlight bash >}}
-ozone sh volume create --quota=1TB --user=bilbo /hive
+ozone sh volume create -ssq=1TB --user=bilbo /hive
Review comment:
In design doc, it also provide a full name called ssQuota, Not sure if there are other better names, or we can use 'spaceQuota' instead, but I think this can be explained in document.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r437116560
##########
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##########
@@ -346,6 +346,9 @@ message VolumeInfo {
optional uint64 creationTime = 7;
optional uint64 objectID = 8;
optional uint64 updateID = 9;
+ optional uint64 quotaUsageInBytes = 10;
+ optional uint64 quotaInCounts = 11;
+ optional uint64 quotaUsageInCounts = 12;
Review comment:
Will it be better if we can enhance `OzoneQuota` to let it not only handle quota in bytes but also quota in counts, such that encapsulate code in `TraditionalBinaryPrefix`?
It would be cleaner by adding a OzoneQuotaProto to encapsulate quotaUsageInBytes, quotaUsageInCounts, quotaUsageInCount, quotaInCount, and reuse it.
BTW, `namespaceQuota`, `storagespaceQuota`, `ssQuota`, `nsQuota`, maybe we need unify the naming.
##########
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##########
@@ -346,6 +346,9 @@ message VolumeInfo {
optional uint64 creationTime = 7;
optional uint64 objectID = 8;
optional uint64 updateID = 9;
+ optional uint64 quotaUsageInBytes = 10;
+ optional uint64 quotaInCounts = 11;
+ optional uint64 quotaUsageInCounts = 12;
Review comment:
Will it be better if we can enhance `OzoneQuota` to let it not only handle quota in bytes but also quota in counts, such that encapsulate code in `TraditionalBinaryPrefix`?
It would be cleaner by adding a `OzoneQuotaProto` to encapsulate `quotaUsageInBytes`, `quotaUsageInCounts`, `quotaUsageInCount`, `quotaInCount`, and reuse it.
BTW, `namespaceQuota`, `storagespaceQuota`, `ssQuota`, `nsQuota`, maybe we need unify the naming.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 closed pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 closed pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] GlenGeng commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
GlenGeng commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r437846415
##########
File path: hadoop-ozone/client/src/main/java/org/apache/hadoop/ozone/client/VolumeArgs.java
##########
@@ -107,7 +118,8 @@ public String getQuota() {
public static class Builder {
private String adminName;
private String ownerName;
- private String volumeQuota;
+ private String volumeSsQuota;
+ private long volumeNsQuota;
Review comment:
`volumeSsQuota` and `volumeNsQuota`
ditto, consider a better naming, it will be easier to for others to use and maintain.
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/OzoneQuota.java
##########
@@ -25,26 +25,73 @@
* represents an OzoneQuota Object that can be applied to
* a storage volume.
*/
-public class OzoneQuota {
+public final class OzoneQuota {
public static final String OZONE_QUOTA_BYTES = "BYTES";
+ public static final String OZONE_QUOTA_KB = "KB";
public static final String OZONE_QUOTA_MB = "MB";
public static final String OZONE_QUOTA_GB = "GB";
public static final String OZONE_QUOTA_TB = "TB";
- private Units unit;
- private long size;
-
/** Quota Units.*/
public enum Units {UNDEFINED, BYTES, KB, MB, GB, TB}
+ private long namespaceQuota;
Review comment:
add comments to explain the meaning of namespaceQuota and storagespaceQuota.
##########
File path: hadoop-hdds/docs/content/shell/VolumeCommands.md
##########
@@ -42,7 +42,7 @@ assign it to a user.
| Uri | The name of the volume. |
{{< highlight bash >}}
-ozone sh volume create --quota=1TB --user=bilbo /hive
+ozone sh volume create -ssq=1TB --user=bilbo /hive
Review comment:
`-ssq `is hard to understand, thus hard to use. Can we give it a self explain name ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r454306051
##########
File path: hadoop-hdds/docs/content/shell/VolumeCommands.md
##########
@@ -42,7 +42,7 @@ assign it to a user.
| Uri | The name of the volume. |
{{< highlight bash >}}
-ozone sh volume create --quota=1TB --user=bilbo /hive
+ozone sh volume create -ssq=1TB --user=bilbo /hive
Review comment:
Here we can refer to the naming rules of the [HDFS command](https://hadoop.apache.org/docs/r2.7.2/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html), in HDFS:
The quota is a hard limit on the number of file.
The space quota is a hard limit on the number of bytes used by files.
So spaceQuota can be used here.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] elek commented on pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
elek commented on pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#issuecomment-651672160
/pending comments from @timmylicheng are not addressed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r441407487
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/OzoneQuota.java
##########
@@ -25,26 +25,75 @@
* represents an OzoneQuota Object that can be applied to
* a storage volume.
*/
-public class OzoneQuota {
+public final class OzoneQuota {
public static final String OZONE_QUOTA_BYTES = "BYTES";
+ public static final String OZONE_QUOTA_KB = "KB";
public static final String OZONE_QUOTA_MB = "MB";
public static final String OZONE_QUOTA_GB = "GB";
public static final String OZONE_QUOTA_TB = "TB";
- private Units unit;
- private long size;
-
/** Quota Units.*/
public enum Units {UNDEFINED, BYTES, KB, MB, GB, TB}
+ // Quota to decide how many buckets or keys can be created.
+ private long namespaceQuota;
+ // Quota to decide how many storage space will be used in bytes.
+ private long storagespaceQuota;
+ private RawStorageSpaceQuota rawStoragespaceQuota;
+
+ private static class RawStorageSpaceQuota {
+ private Units unit;
+ private long size;
+
+ RawStorageSpaceQuota(Units unit, long size) {
+ this.unit = unit;
+ this.size = size;
+ }
+
+ public Units getUnit() {
+ return unit;
+ }
+
+ public long getSize() {
+ return size;
+ }
+
+ /**
+ * Returns size in Bytes or -1 if there is no Quota.
+ */
+ public long sizeInBytes() {
Review comment:
OK, I will review OzoneConsts again and see what can be used.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] Simon0806 commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
Simon0806 commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r437142895
##########
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##########
@@ -346,6 +346,9 @@ message VolumeInfo {
optional uint64 creationTime = 7;
optional uint64 objectID = 8;
optional uint64 updateID = 9;
+ optional uint64 quotaUsageInBytes = 10;
+ optional uint64 quotaInCounts = 11;
+ optional uint64 quotaUsageInCounts = 12;
Review comment:
That's a good point, I will see the effect to modify.
##########
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##########
@@ -346,6 +346,9 @@ message VolumeInfo {
optional uint64 creationTime = 7;
optional uint64 objectID = 8;
optional uint64 updateID = 9;
+ optional uint64 quotaUsageInBytes = 10;
+ optional uint64 quotaInCounts = 11;
+ optional uint64 quotaUsageInCounts = 12;
Review comment:
That's a good point, I will see the workload to modify.
##########
File path: hadoop-ozone/common/src/main/proto/OzoneManagerProtocol.proto
##########
@@ -346,6 +346,9 @@ message VolumeInfo {
optional uint64 creationTime = 7;
optional uint64 objectID = 8;
optional uint64 updateID = 9;
+ optional uint64 quotaUsageInBytes = 10;
+ optional uint64 quotaInCounts = 11;
+ optional uint64 quotaUsageInCounts = 12;
Review comment:
That's a good point, but here I just want to extend it, if I add a OzoneQuotaProto to encapsulate all the quota related fields, the quotaInBytes will be useless in original pb format.
I will check if there are some places can be unify the variable name.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] timmylicheng commented on a change in pull request #1041: HDDS-3725. Ozone sh volume client support quota option
Posted by GitBox <gi...@apache.org>.
timmylicheng commented on a change in pull request #1041:
URL: https://github.com/apache/hadoop-ozone/pull/1041#discussion_r440104690
##########
File path: hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientAbstract.java
##########
@@ -255,10 +255,9 @@ public void testSetVolumeQuota()
throws IOException {
String volumeName = UUID.randomUUID().toString();
store.createVolume(volumeName);
- store.getVolume(volumeName).setQuota(
- OzoneQuota.parseQuota("100000000 BYTES"));
+ store.getVolume(volumeName).setQuota(OzoneQuota.parseQuota("1GB", 0L));
OzoneVolume volume = store.getVolume(volumeName);
- Assert.assertEquals(100000000L, volume.getQuota());
+ Assert.assertEquals(1073741824L, volume.getStoragespaceQuota());
Review comment:
how about some number * 1024?
##########
File path: hadoop-hdds/docs/content/shell/VolumeCommands.md
##########
@@ -42,7 +42,7 @@ assign it to a user.
| Uri | The name of the volume. |
{{< highlight bash >}}
-ozone sh volume create --quota=1TB --user=bilbo /hive
+ozone sh volume create -ssq=1TB --user=bilbo /hive
Review comment:
I would prefer quota in admin command IMO
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/client/OzoneQuota.java
##########
@@ -25,26 +25,75 @@
* represents an OzoneQuota Object that can be applied to
* a storage volume.
*/
-public class OzoneQuota {
+public final class OzoneQuota {
public static final String OZONE_QUOTA_BYTES = "BYTES";
+ public static final String OZONE_QUOTA_KB = "KB";
public static final String OZONE_QUOTA_MB = "MB";
public static final String OZONE_QUOTA_GB = "GB";
public static final String OZONE_QUOTA_TB = "TB";
- private Units unit;
- private long size;
-
/** Quota Units.*/
public enum Units {UNDEFINED, BYTES, KB, MB, GB, TB}
+ // Quota to decide how many buckets or keys can be created.
+ private long namespaceQuota;
+ // Quota to decide how many storage space will be used in bytes.
+ private long storagespaceQuota;
+ private RawStorageSpaceQuota rawStoragespaceQuota;
+
+ private static class RawStorageSpaceQuota {
+ private Units unit;
+ private long size;
+
+ RawStorageSpaceQuota(Units unit, long size) {
+ this.unit = unit;
+ this.size = size;
+ }
+
+ public Units getUnit() {
+ return unit;
+ }
+
+ public long getSize() {
+ return size;
+ }
+
+ /**
+ * Returns size in Bytes or -1 if there is no Quota.
+ */
+ public long sizeInBytes() {
Review comment:
Why not just use OzoneConsts?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org