You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/05/10 06:41:35 UTC
[GitHub] [cloudstack] Pearl1594 opened a new pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Pearl1594 opened a new pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005
### Description
This PR ensure consistency between usage details stored across different resource types. For the sake of consistency, backup usage record will show the time duration of backup usage in the raw_usage / usage_display fields.
<!--- ********************************************************************************* -->
<!--- NOTE: AUTOMATATION USES THE DESCRIPTIONS TO SET LABELS AND PRODUCE DOCUMENTATION. -->
<!--- PLEASE PUT AN 'X' in only **ONE** box -->
<!--- ********************************************************************************* -->
### Types of changes
- [ ] Breaking change (fix or feature that would cause existing functionality to change)
- [ ] New feature (non-breaking change which adds functionality)
- [X] Bug fix (non-breaking change which fixes an issue)
- [ ] Enhancement (improves an existing feature and functionality)
- [ ] Cleanup (Code refactoring and cleanup, that may add test cases)
### Feature/Enhancement Scale or Bug Severity
#### Bug Severity
- [ ] BLOCKER
- [ ] Critical
- [ ] Major
- [X] Minor
- [ ] Trivial
### Screenshots (if appropriate):
### How Has This Been Tested?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-890670577
Hi @Pearl1594, @weizhouapache has created PR #5259 to fix #4982, can you please sync to work on a unified solution?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-836520432
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
nvazquez commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-896467114
Hi @Pearl1594 any update on this? Can both PRs be integrated into a single one to fix the issue #4982?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909033822
@Pearl1594 tentatively moving to 4.15.2.0 milestone - please advise if it's feasible/ready for 4.15?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-836508006
@blueorangutan package
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 closed pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 closed pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909035481
Since PR #5259 addresses the same issue - I'll close this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909035481
Since PR #5259 addresses the same issue - I'll close this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-836521093
@Pearl1594 a Jenkins job has been kicked to build packages. I'll keep you posted as I make progress.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] blueorangutan commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
blueorangutan commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-836644039
Packaging result: :heavy_check_mark: centos7 :heavy_multiplication_x: centos8 :heavy_multiplication_x: debian. SL-JID 554
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 closed pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 closed pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909035481
Since PR #5259 addresses the same issue - I'll close this PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-890683885
Will do @nvazquez @weizhouapache - Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909033822
@Pearl1594 tentatively moving to 4.15.2.0 milestone - please advise if it's feasible/ready for 4.15?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 closed pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 closed pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] GutoVeronezi commented on a change in pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
GutoVeronezi commented on a change in pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#discussion_r629437113
##########
File path: usage/src/main/java/com/cloud/usage/parser/BackupUsageParser.java
##########
@@ -68,38 +70,91 @@ public static boolean parse(AccountVO account, Date startDate, Date endDate) {
return true;
}
- final Map<Long, BackupInfo> vmUsageMap = new HashMap<>();
+ Map<String, Pair<Long, Long>> bkpUsageMap = new HashMap<String, Pair<Long, Long>>();
+ final Map<String, BackupInfo> vmUsageMap = new HashMap<>();
for (final UsageBackupVO usageBackup : usageBackups) {
final Long vmId = usageBackup.getVmId();
final Long zoneId = usageBackup.getZoneId();
+ String key = vmId + "-" + usageBackup.getBackupOfferingId() + "-" + zoneId;
final Long offeringId = usageBackup.getBackupOfferingId();
- if (vmUsageMap.get(vmId) == null) {
- vmUsageMap.put(vmId, new BackupUsageParser.BackupInfo(new Backup.Metric(0L, 0L), zoneId, vmId, offeringId));
+ if (vmUsageMap.get(key) == null) {
+ vmUsageMap.put(key, new BackupUsageParser.BackupInfo(new Backup.Metric(0L, 0L), zoneId, vmId, offeringId));
}
- final Backup.Metric metric = vmUsageMap.get(vmId).getMetric();
+ final Backup.Metric metric = vmUsageMap.get(key).getMetric();
metric.setBackupSize(metric.getBackupSize() + usageBackup.getSize());
metric.setDataSize(metric.getDataSize() + usageBackup.getProtectedSize());
+ long duration = getUsageDuration(usageBackup, startDate, endDate);
+ updateBackupUsageData(bkpUsageMap, key, vmId, duration);
}
- for (final BackupInfo backupInfo : vmUsageMap.values()) {
- final Long vmId = backupInfo.getVmId();
- final Long zoneId = backupInfo.getZoneId();
- final Long offeringId = backupInfo.getOfferingId();
- final Double rawUsage = (double) backupInfo.getMetric().getBackupSize();
- final Double sizeGib = rawUsage / (1024.0 * 1024.0 * 1024.0);
- final String description = String.format("Backup usage VM ID: %d", vmId);
- final String usageDisplay = String.format("%.4f GiB", sizeGib);
-
- final UsageVO usageRecord =
- new UsageVO(zoneId, account.getAccountId(), account.getDomainId(), description, usageDisplay,
- UsageTypes.BACKUP, rawUsage, vmId, null, offeringId, null, vmId,
- backupInfo.getMetric().getBackupSize(), backupInfo.getMetric().getDataSize(), startDate, endDate);
- s_usageDao.persist(usageRecord);
+ for (String bkpIdKey : bkpUsageMap.keySet()) {
+ Pair<Long, Long> bkptimeInfo = bkpUsageMap.get(bkpIdKey);
+ long useTime = bkptimeInfo.second();
+
+ // Only create a usage record if we have a runningTime of bigger than zero.
+ if (useTime > 0L) {
+ BackupUsageParser.BackupInfo info = vmUsageMap.get(bkpIdKey);
+ createUsageRecord(UsageTypes.BACKUP, account, useTime, startDate, endDate, info);
+ }
}
return true;
}
+ private static void createUsageRecord(int type, AccountVO account, long runningTime, Date startDate, Date endDate, BackupInfo backupInfo) {
+ // Our smallest increment is hourly for now
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("Total running time " + runningTime + "ms");
+ }
+ final Long vmId = backupInfo.getVmId();
+ final Long zoneId = backupInfo.getZoneId();
+ final Long offeringId = backupInfo.getOfferingId();
+ float usage = runningTime / 1000f / 60f / 60f;
Review comment:
@Pearl1594 as this calculation is done several times in the code, we could create and utility to it.
##########
File path: usage/src/main/java/com/cloud/usage/parser/BackupUsageParser.java
##########
@@ -68,38 +70,91 @@ public static boolean parse(AccountVO account, Date startDate, Date endDate) {
return true;
}
- final Map<Long, BackupInfo> vmUsageMap = new HashMap<>();
+ Map<String, Pair<Long, Long>> bkpUsageMap = new HashMap<String, Pair<Long, Long>>();
+ final Map<String, BackupInfo> vmUsageMap = new HashMap<>();
for (final UsageBackupVO usageBackup : usageBackups) {
final Long vmId = usageBackup.getVmId();
final Long zoneId = usageBackup.getZoneId();
+ String key = vmId + "-" + usageBackup.getBackupOfferingId() + "-" + zoneId;
final Long offeringId = usageBackup.getBackupOfferingId();
- if (vmUsageMap.get(vmId) == null) {
- vmUsageMap.put(vmId, new BackupUsageParser.BackupInfo(new Backup.Metric(0L, 0L), zoneId, vmId, offeringId));
+ if (vmUsageMap.get(key) == null) {
+ vmUsageMap.put(key, new BackupUsageParser.BackupInfo(new Backup.Metric(0L, 0L), zoneId, vmId, offeringId));
}
- final Backup.Metric metric = vmUsageMap.get(vmId).getMetric();
+ final Backup.Metric metric = vmUsageMap.get(key).getMetric();
metric.setBackupSize(metric.getBackupSize() + usageBackup.getSize());
metric.setDataSize(metric.getDataSize() + usageBackup.getProtectedSize());
+ long duration = getUsageDuration(usageBackup, startDate, endDate);
+ updateBackupUsageData(bkpUsageMap, key, vmId, duration);
}
- for (final BackupInfo backupInfo : vmUsageMap.values()) {
- final Long vmId = backupInfo.getVmId();
- final Long zoneId = backupInfo.getZoneId();
- final Long offeringId = backupInfo.getOfferingId();
- final Double rawUsage = (double) backupInfo.getMetric().getBackupSize();
- final Double sizeGib = rawUsage / (1024.0 * 1024.0 * 1024.0);
- final String description = String.format("Backup usage VM ID: %d", vmId);
- final String usageDisplay = String.format("%.4f GiB", sizeGib);
-
- final UsageVO usageRecord =
- new UsageVO(zoneId, account.getAccountId(), account.getDomainId(), description, usageDisplay,
- UsageTypes.BACKUP, rawUsage, vmId, null, offeringId, null, vmId,
- backupInfo.getMetric().getBackupSize(), backupInfo.getMetric().getDataSize(), startDate, endDate);
- s_usageDao.persist(usageRecord);
+ for (String bkpIdKey : bkpUsageMap.keySet()) {
+ Pair<Long, Long> bkptimeInfo = bkpUsageMap.get(bkpIdKey);
+ long useTime = bkptimeInfo.second();
+
+ // Only create a usage record if we have a runningTime of bigger than zero.
+ if (useTime > 0L) {
+ BackupUsageParser.BackupInfo info = vmUsageMap.get(bkpIdKey);
+ createUsageRecord(UsageTypes.BACKUP, account, useTime, startDate, endDate, info);
+ }
}
return true;
}
+ private static void createUsageRecord(int type, AccountVO account, long runningTime, Date startDate, Date endDate, BackupInfo backupInfo) {
+ // Our smallest increment is hourly for now
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("Total running time " + runningTime + "ms");
+ }
+ final Long vmId = backupInfo.getVmId();
+ final Long zoneId = backupInfo.getZoneId();
+ final Long offeringId = backupInfo.getOfferingId();
+ float usage = runningTime / 1000f / 60f / 60f;
Review comment:
@Pearl1594 as this calculation is done several times in the code, we could create an utility to it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
rhtyd commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-909033822
@Pearl1594 tentatively moving to 4.15.2.0 milestone - please advise if it's feasible/ready for 4.15?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-889681141
@Pearl1594 does this pr fix #4982 ? is it ready for review ?
@nvazquez
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on pull request #5005: usage backup: Raw usage/usage display to record time duration of usage
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on pull request #5005:
URL: https://github.com/apache/cloudstack/pull/5005#issuecomment-896470785
Hi @nvazquez - was held up in other tasks, hence wasn't able to test #5259. However, on doing a code review, I believe, both PRs solve the same issue, and @weizhouapache's solution would be less disruptive and more apt to address the issue - so maybe we could go ahead with #5259. I'll spend some time testing it as well. Apologies for the delay.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org