You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by GitBox <gi...@apache.org> on 2020/09/10 11:02:45 UTC
[GitHub] [hadoop-ozone] captainzmc opened a new pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
captainzmc opened a new pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337
## What changes were proposed in this pull request?
Currently MAX_QUOTA_IN_BYTES is 1EB, which is not very exact. Later we need use this to determine whether quota is enabled or not.
We need to change it to long.max_value to be [consistent with HDFS](https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsQuotaAdminGuide.html#Space_Quotas).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-4129
## How was this patch tested?
Existing UT
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483493420
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
Review comment:
Yes, the corresponding [UT has been added](https://github.com/apache/hadoop-ozone/pull/1337/files#diff-d914f1d4b911db08ba389a30c322a338R308).
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r476253156
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
Thanks for @adoroszlai 's discussion. Yes, the previous volume will enable quota. So I created JIRA HDDS-4106, which supports user Clearing spaceQuota.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-679873338
> Closing this as [HDDS-4089](https://issues.apache.org/jira/browse/HDDS-4089) planning to fix this issue to make ACL commands work with the external authorizer.
Hi @bharatviswa504 , I think this change has nothing to do with ACL commands. Is it something I don't understand?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r477121185
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
I double checked HDFS implementation，it doesn't use LONG.MAX_VALUE to verify whether quota is enabled or not. It use "-1".
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r477121185
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
I double checked HDFS implementation，it doesn't use LONG.MAX_VALUE to verify whether quota is enabled or not. It use "-1" which is more reasonable for a set/unset check.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483501073
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
Review comment:
This judgment is redundant, will delete it.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc closed pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc closed pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r475697694
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
What about volumes created before this change with the previous max. value? Will those be treated by new version as if quota had been enabled for them?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-690919896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r478042483
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
Thanks sammi for the discussion. It is indeed uncommon to use the maximum value to determine whether to enable quota, and it is more common to use -1 to determine. Later I will confirm whether to enable quota with -1 in HDDS-3727. The MAX_QUOTA_IN_BYTES here will be used to determine if the parameter values for setquota and updatequota are reasonable in HDDS-3725.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-686945316
Hi @ChenSammi @adoroszlai , I had added the initialization and change the old volume quota default value to -1 when OM starts. Could you help review this PR again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r477014939
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
Suggest add new logic during OM startup to reset existing Volume max quota value to mitigate the compatibility issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-690919896
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] adoroszlai commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
adoroszlai commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-679893004
> > Closing this as [HDDS-4089](https://issues.apache.org/jira/browse/HDDS-4089) planning to fix this issue to make ACL commands work with the external authorizer.
>
> Hi @bharatviswa504 , I think this change has nothing to do with ACL commands. Is it something I don't understand?
@captainzmc I think you can ignore it, the comment was intended for another PR: https://github.com/apache/hadoop-ozone/pull/1271#issuecomment-679274161
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483480047
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
Review comment:
Can we add a UT for this function?
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
+ omVolumeArgs.setQuotaInBytes(-1);
+ // Commit to DB.
+ BatchOperation batchOperation =
+ metadataManager.getStore().initBatchOperation();
+ metadataManager.getVolumeTable().putWithBatch(batchOperation,
+ omVolumeKey, omVolumeArgs);
+ metadataManager.getStore().commitBatchOperation(batchOperation);
+
+ // Add to cache.
+ long transactionID = (Long.MAX_VALUE - 1) >> 8;
+ metadataManager.getVolumeTable().addCacheEntry(
+ new CacheKey<>(omVolumeKey),
+ new CacheValue<>(Optional.of(omVolumeArgs), transactionID));
+ LOG.info("Update Volume {}'s quota to -1, , default does not enable" +
Review comment:
duplicate ","
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
There is one risk here, if user set the quota to 1EB specifically, we cann't reset it to -1 on next OM restart.
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
+ omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES) {
Review comment:
Why omVolumeArgs.getQuotaInBytes() == OzoneConsts.MAX_QUOTA_IN_BYTES check is required here?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-680568252
Sorry, @captainzmc I have posted the comment on the wrong PR. I was trying to search where I have posted, not able to figure out. You can ignore my comment.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483510372
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
Currently, the way to determine the old quota is by seeing whether his quota is 1 EB. So 1EB is still a special value. At present, OZone's cluster size is not large, far from the upper limit of 1EB.
So can we ignore this case? In addition, I adjust the LOG level from INFO to Warn (I can change it to ERROR if needed) to alert the user that the corresponding quota has been changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r478042483
##########
File path: hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/OzoneConsts.java
##########
@@ -200,9 +200,9 @@ public static Versioning getVersioning(boolean versioning) {
/**
- * Max OM Quota size of 1024 PB.
+ * Max OM Quota size of Long.MAX_VALUE.
*/
- public static final long MAX_QUOTA_IN_BYTES = 1024L * 1024 * TB;
+ public static final long MAX_QUOTA_IN_BYTES = Long.MAX_VALUE;
Review comment:
Thanks sammi for the discussion. It is indeed uncommon to use the maximum value to determine whether to enable quota, and it is more common to use -1 to determine. Later I will confirm whether to enable quota with -1 in HDDS-3727. The MAX_QUOTA_IN_BYTES here will be used to determine if the parameter values for setquota and updatequota are reasonable.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-690919896
+1. After a thoroughly thinking, we choose a conservative way.
Thanks @captainzmc for the contribution and @adoroszlai for the review.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc edited a comment on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc edited a comment on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-686945316
Hi @ChenSammi , I had added the initialization and change the old volume quota default value to -1 when OM starts. Could you help review this PR again?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-688174746
Thanks for Sammi's review, fixed review issues. @ChenSammi Could you help take another look?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] ChenSammi merged pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
ChenSammi merged pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] captainzmc commented on a change in pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
captainzmc commented on a change in pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#discussion_r483510372
##########
File path: hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java
##########
@@ -3540,6 +3546,43 @@ public boolean getEnableFileSystemPaths() {
OZONE_OM_ENABLE_FILESYSTEM_PATHS_DEFAULT);
}
+ /**
+ * Initialize volume's quota, which resolves the problem that the old quota
+ * initial value was not set correctly. This method will be removed in a
+ * later version.
+ * @throws IOException
+ */
+ private void initializeOldVolumeQuota() throws IOException {
+
+ TableIterator<String, ? extends Table.KeyValue<String,
+ OmVolumeArgs>> iterator = metadataManager.getVolumeTable().iterator();
+ while(iterator.hasNext()) {
+ OmVolumeArgs omVolumeArgs = iterator.next().getValue();
+ String omVolumeKey = metadataManager.getVolumeKey(
+ omVolumeArgs.getVolume());
+ // Previously, the volume quota was created to default to 1EB. To change
+ // to -1, default does not enable quota.
+ if (omVolumeArgs.getQuotaInBytes() == 1024 * 1024 * TB ||
Review comment:
Currently, the way to determine the old quota is by seeing whether his quota is 1 EB. So 1EB is still a special value. At present, OZone's cluster size is not large, far from the upper limit of 1EB.
So can we ignore this case? In addition, I adjust the LOG level from INFO to Warn (I can change it to error if needed) to alert the user that the corresponding quota has been changed.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org
[GitHub] [hadoop-ozone] bharatviswa504 commented on pull request #1337: HDDS-4129. change MAX_QUOTA_IN_BYTES to Long.MAX_VALUE.
Posted by GitBox <gi...@apache.org>.
bharatviswa504 commented on pull request #1337:
URL: https://github.com/apache/hadoop-ozone/pull/1337#issuecomment-679223880
Closing this as HDDS-4089 planning to fix this issue to make ACL commands work with the external authorizer.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: ozone-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: ozone-issues-help@hadoop.apache.org