You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@ozone.apache.org by "whbing (via GitHub)" <gi...@apache.org> on 2023/08/14 11:18:56 UTC
[GitHub] [ozone] whbing opened a new pull request, #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
whbing opened a new pull request, #5186:
URL: https://github.com/apache/ozone/pull/5186
## What changes were proposed in this pull request?
The default value in `QuotaRepairUpgradeAction` class is not the same in `ozone-site.xml`, which is confusing.
In fact, `QuotaRepairTask` will be very time consuming during upgrading om, the time taken is as follows:
```java
// test 1
2023-07-05 16:15:36,432 [pool-37-thread-1] INFO org.apache.hadoop.ozone.om.service.QuotaRepairTask: Recalculate Key usages completed, count 259337075 time 1873737ms
// test 2
2023-08-13 17:35:45,622 [pool-37-thread-2] INFO org.apache.hadoop.ozone.om.service.QuotaRepairTask: Recalculate File usages completed, count 19752234 time 440393ms
```
I prefer that `ozone.om.upgrade.quota.recalculate.enabled` default to false.
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-9166
## How was this patch tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1680266953
@whbing , can you do a master rebase? There is one compile issued fixed in master branch recently.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1693356558
I will add new JIRA to add this in documentation ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1680507351
> @whbing , can you do a master rebase? There is one compile issued fixed in master branch recently.
Thanks! Already rebased latest master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1735006699
> @whbing Plz rebase the PR, there are few random issue fixed in upstream
Rebased from master. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1692949110
> > @ChenSammi @whbing I have a query,
> >
> > * If we disable by default, then usages count will not be correct, and hence quota if used may not work properly. Shall we still disable?
>
> Enable by default might lead users to experience unexpectedly long waiting times. Especially when there is no quota setting, the recalculation is not necessary. I think disable by default might be even more user-friendly.
@whbing The Quota setting is limiting the quota usages. But usages information itself is not correct, then user can not make use of this feature correctly later on.
So there is a functional issue in this case. Based on this point, please check user experience for upgrade case.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1692998638
> @whbing The Quota setting is limiting the quota usages. But usages information itself is not correct, then user can not make use of this feature correctly later on.
>
> So there is a functional issue in this case. Based on this point, please check user experience for upgrade case.
@sumitagrawl Thanks for explaining.
By the way, should we consider adding a description about `recalculation-quota` in the "Nonrolling-Upgrade.md" document if enabled default?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1692731347
> > @whbing , can you do a master rebase? There is one compile issued fixed in master branch recently.
>
> @ChenSammi @whbing I have a query,
>
> * If we disable by default, then usages count will not be correct, and hence quota if used may not work properly. Shall we still disable?
@sumitagrawl , it's important and great that we provide this usage recalculation to all existing users. Due to its potential long execution time(during this time, the cluster cannot provide services) I think we also should let user aware of that and opt-in to run the task proactively. I think that's this JIRA's motivation. @whbing , you can add more here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1677144979
I attempted to increase the value of `QuotaRepairTask#TASK_THREAD_CNT` and there was no noticeable improvement. In a situation with such a large number of files, disabling QuotaRepairTask would be more efficient.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on a diff in pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on code in PR #5186:
URL: https://github.com/apache/ozone/pull/5186#discussion_r1308354092
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/QuotaRepairUpgradeAction.java:
##########
@@ -35,7 +35,8 @@ public class QuotaRepairUpgradeAction implements OmUpgradeAction {
@Override
public void execute(OzoneManager arg) throws Exception {
boolean enabled = arg.getConfiguration().getBoolean(
- OMConfigKeys.OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE, false);
Review Comment:
> yes, we can set default as true. I was thinking if not configured, then go with false, but this can be changed.
@sumitagrawl `true` in ozone-default.xml will take effect if ozone-site.xml not configured. So, I just set the value of OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE to true to avoid misleading.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1696932671
After the above discussion, the default value is set to `true`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1677149084
@sumitagrawl PTAL, thanks !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl merged pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl merged PR #5186:
URL: https://github.com/apache/ozone/pull/5186
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on a diff in pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on code in PR #5186:
URL: https://github.com/apache/ozone/pull/5186#discussion_r1305378866
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/QuotaRepairUpgradeAction.java:
##########
@@ -35,7 +35,8 @@ public class QuotaRepairUpgradeAction implements OmUpgradeAction {
@Override
public void execute(OzoneManager arg) throws Exception {
boolean enabled = arg.getConfiguration().getBoolean(
- OMConfigKeys.OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE, false);
Review Comment:
In addition, the default value should be consistent with ozone-default.xml
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1689587448
> @whbing , can you do a master rebase? There is one compile issued fixed in master branch recently.
@ChenSammi @whbing I have a query,
- If we disable by default, then usages count will not be correct, and hence quota if used may not work properly. Shall we still disable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1734931339
@whbing Plz rebase the PR, there are few random issue fixed in upstream
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] ChenSammi commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "ChenSammi (via GitHub)" <gi...@apache.org>.
ChenSammi commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1680265457
I'm kind agree that it's better to set the default as false. For any user, who want to do a recalculation, he/she can turn on the switch explicitly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] whbing commented on pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "whbing (via GitHub)" <gi...@apache.org>.
whbing commented on PR #5186:
URL: https://github.com/apache/ozone/pull/5186#issuecomment-1692925876
>
> @ChenSammi @whbing I have a query,
>
> * If we disable by default, then usages count will not be correct, and hence quota if used may not work properly. Shall we still disable?
Enable by default might lead users to experience unexpectedly long waiting times.
Especially when there is no quota setting, the recalculation is not necessary.
I think disable by default might be even more user-friendly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org
[GitHub] [ozone] sumitagrawl commented on a diff in pull request #5186: HDDS-9166. Fix OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE default value
Posted by "sumitagrawl (via GitHub)" <gi...@apache.org>.
sumitagrawl commented on code in PR #5186:
URL: https://github.com/apache/ozone/pull/5186#discussion_r1305654778
##########
hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/upgrade/QuotaRepairUpgradeAction.java:
##########
@@ -35,7 +35,8 @@ public class QuotaRepairUpgradeAction implements OmUpgradeAction {
@Override
public void execute(OzoneManager arg) throws Exception {
boolean enabled = arg.getConfiguration().getBoolean(
- OMConfigKeys.OZONE_OM_UPGRADE_QUOTA_RECALCULATE_ENABLE, false);
Review Comment:
yes, we can set default as true. I was thinking if not configured, then go with false, but this can be changed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@ozone.apache.org
For additional commands, e-mail: issues-help@ozone.apache.org