You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/04/30 15:32:24 UTC
[GitHub] [cloudstack] olivierlemasle opened a new issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
olivierlemasle opened a new issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
Usage
~~~
##### CLOUDSTACK VERSION
~~~
4.14.1.0
4.15.0.0
~~~
##### CONFIGURATION
Backup framework enabled
##### OS / ENVIRONMENT
N/A
##### SUMMARY
When a VM is removed from a backup offering, then re-assigned to the same backup offering, the usage record generated for the period in which these events occur has an invalid usage value, equal to the sum of the sizes of the backups.
##### STEPS TO REPRODUCE
Example scenario:
1. `02:30 PM`: Create a VM and assign it to a backup offering
2. `02:35 PM`: Make backups of the VM (for a size of e.g. 1 GiB)
3. `03:15 PM`: Remove the backup offering, with the "force" option (to delete existing backups)
4. `03:30 PM`: Re-assign the VM to the backup offering
5. `03:35 PM`: Make a backup of the VM (for a size of e.g. 0.8 GiB)
To reproduce the bug, if the usage aggregation range is e.g 1 Hour (`usage.stats.job.aggregation.range = 60`), make sure the steps 3, 4 and 5 are executed in the same hour.
##### EXPECTED RESULTS
Ideally, I would expect CloudStack usage server to:
- generate two usage records for the period between `03:00 PM` and `04:00 PM` (one for each period with a backup offering)
- use the fields `raw_usage` / `usage_display` to show the duration of usage and the field `size` to show the size of the backups (that's the way storage usages are generated for volumes, templates, isos and snashots)
```
+------------+---------------+------------+------------+----------+----------+------------------------+
| raw_usage | usage_display | size | start_date | end_date | usage_id | description |
+------------+---------------+------------+------------+----------+----------+------------------------+
| 0.5 | 0.5 Hrs | 1073741824 | 14:00:00 | 14:59:59 | 67 | Backup usage VM ID: 67 |
| 0.25 | 0.25 Hrs | 1073741824 | 15:00:00 | 15:59:59 | 67 | Backup usage VM ID: 67 |
| 0.5 | 0.5 Hrs | 858993460 | 15:00:00 | 15:59:59 | 67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+
```
This could also be ok (even if we loose information):
```
+------------+---------------+------------+------------+----------+----------+------------------------+
| raw_usage | usage_display | size | start_date | end_date | usage_id | description |
+------------+---------------+------------+------------+----------+----------+------------------------+
| 1073741824 | 1.0 GiB | 1073741824 | 14:00:00 | 14:59:59 | 67 | Backup usage VM ID: 67 |
| 858993460 | 0.8 GiB | 858993460 | 15:00:00 | 15:59:59 | 67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+
```
##### ACTUAL RESULTS
Usage records:
```
+------------+---------------+------------+------------+----------+----------+------------------------+
| raw_usage | usage_display | size | start_date | end_date | usage_id | description |
+------------+---------------+------------+------------+----------+----------+------------------------+
| 1073741824 | 1.0 GiB | 1073741824 | 14:00:00 | 14:59:59 | 67 | Backup usage VM ID: 67 |
| 1932735284 | 1.8 GiB | 1932735284 | 15:00:00 | 15:59:59 | 67 | Backup usage VM ID: 67 |
+------------+---------------+------------+------------+----------+----------+------------------------+
```
Here, the `raw_usage` for the period from 3 PM to 4 PM is the sum of the backup sizes, which is irrelevant (the 1 GiB backups are deleted before the creation of the 0.8 GiB).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-833805201
Actually, the bug is to sum the backup sizes over the time frame.
I think that what I've called the **Option 1** (cf above) is the best way to handle backup usages. In that option, the usage value is the duration (in hours) over the time frame. This is **aligned with other storage usage types in CloudStack** (volumes, ISOs, templates, snapshots). For all these usage types, the size of the volume/ISO/template/snapshot is displayed in "size", and the duration is displayed "raw_usage" / "usage_display".
The **Option 2** is less accurate. E.g. it does not show that from 14:00:00 to 14:59:59, the VM was assigned to the backup offering during only half the time. However, this option is also less a breaking change.
It consists of just using as the "size" the last known size over the time frame.
I really think these options should be discussed before we implement any change. I would prefer the Option 1 but I'm interested to hear your opinion / the opinion of the community.
On a side node, you wrote:
> ... before removing the offering from the VM and reassigning **another** backup offering ...
It may not be important, but in my scenario above, the VM was re-assigned to the *same* backup offering. This is not a far-fetched scenario: actually, removing the backup offering and re-assigning the VM to the backup offering is the only way to attach or detach a volume to the VM.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd closed issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-895101427
@weizhouapache I'm not following this, please discuss with @Pearl1594 @shwstppr and the rest of the RM/release team. Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] weizhouapache commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
weizhouapache commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-889685194
@Pearl1594 @rhtyd @nvazquez
I created a draft pr #5259 to fix this issue. since @Pearl1594 had some work in pr #5005 , let's work together for a solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836491292
@olivierlemasle @rhtyd Did some tests on VM instance usage to compare its behavior against the way backup usage currently works and observed the following:
**Note:** The usage aggregation time interval ( `usage.stats.job.aggregation.range`) is set to 5 minutes
#### Case 1: When VM is destroyed before the aggregation time interval:
Usage record is generated for only the time period the VM was present: For example:
```
mysql> select * from usage_vm_instance where usage_type = 1;
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+
| usage_type | zone_id | account_id | vm_instance_id | vm_name | service_offering_id | template_id | hypervisor_type | start_date | end_date | cpu_speed | cpu_cores | memory |
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+
| 1 | 1 | 2 | 3 | test | 1 | 111 | Simulator | 2021-05-10 04:35:23 | 2021-05-10 07:40:34 | NULL | NULL | NULL |
+------------+---------+------------+----------------+---------+---------------------+-------------+-----------------+---------------------+---------------------+-----------+-----------+--------+
Corresponding usage record for that time period:
mysql> select id,zone_id,account_id,usage_display,usage_type,raw_usage,start_date,end_date from cloud_usage where usage_type = 1;
| 219 | 1 | 2 | 0.083334 Hrs | 1 | 0.08333361148834229 | 2021-05-10 07:33:13 | 2021-05-10 07:38:13 | <-- 0.083334 Hrs corresponds to 5mins
| 225 | 1 | 2 | 0.038991 Hrs | 1 | 0.03899139165878296 | 2021-05-10 07:38:13 | 2021-05-10 07:43:13 | <-- VM instance deleted at 7:40:34 ; 0.038992 corresponds to approx. 2mins
+-----+---------+------------+---------------+------------+---------------------+---------------------+---------------------+
```
Same holds true for backups as well, when a backup offering is removed, usage record is generated only for the specific duration of the time-interval (aggregation interval) - https://github.com/apache/cloudstack/pull/5005 handles displaying duration of usage in the raw_usage / usage_display fields of the cloud_usage table as opposed to size
#### Case 2: When the VM is scaled i.e., service offering is changed, 2 records are generated
2 records are generated for the time interval when the compute offering was changed
```
mysql> select vm_name,service_offering_id,start_date,end_date from usage_vm_instance where usage_type = 1 and vm_instance_id = 6;
+---------+---------------------+---------------------+---------------------+
| vm_name | service_offering_id | start_date | end_date |
+---------+---------------------+---------------------+---------------------+
| iso1 | 1 | 2021-05-10 08:00:54 | 2021-05-10 08:06:12 |
| iso1 | 2 | 2021-05-10 08:06:12 | NULL |
+---------+---------------------+---------------------+---------------------+
Corresponding usage record when service offering was changed
mysql> select usage_display,raw_usage,start_date,end_date,vm_name,offering_id from cloud_usage where usage_type = 1;
| 0.038787 Hrs | 0.03878694400191307 | 2021-05-10 07:58:13 | 2021-05-10 08:03:13 | iso1 | 1 |
| 0.033787 Hrs | 0.033786945044994354 | 2021-05-10 08:03:13 | 2021-05-10 08:08:13 | iso1 | 2 | <--- 0.033787 Hrs corresponds to 2mins ; @ 8:06 the Service offering was changed hence 2 records generated within the time interval of 8:03 - 8:08
| 0.049547 Hrs | 0.04954666644334793 | 2021-05-10 08:03:13 | 2021-05-10 08:08:13 | iso1 | 1 | <---- 0.049547 Hrs corresponds to 3 mins
```
Case 2 holds true for backups as well, i.e., if the backup offering is changed to another one within the stipulated aggregation interval, 2 records are generated for that time period.
At `8:45` the backup offering was unassigned from a VM and a **new** backup offering was assigned to it:
```
mysql> select vm_id,backup_offering_id,created,removed from usage_backup where vm_id =6;
+-------+--------------------+---------------------+---------------------+
| vm_id | backup_offering_id | created | removed |
+-------+--------------------+---------------------+---------------------+
| 6 | 1 | 2021-05-10 08:34:05 | 2021-05-10 08:45:27 | <-- Backup Offering removed
| 6 | 2 | 2021-05-10 08:45:38 | NULL | <-- New Backup offering assigned to the VM
+-------+--------------------+---------------------+---------------------+
mysql> select id,offering_id,usage_display,raw_usage,size,start_date,end_date from cloud_usage where usage_type = 28 and vm_instance_id = 6;
+-----+-------------+---------------+----------------------+------+---------------------+---------------------+
| id | offering_id | usage_display | raw_usage | size | start_date | end_date |
+-----+-------------+---------------+----------------------+------+---------------------+---------------------+
| 296 | 1 | 0.069064 Hrs | 0.06906416267156601 | 0 | 2021-05-10 08:33:13 | 2021-05-10 08:38:13 |
| 303 | 1 | 0.083329 Hrs | 0.08332943916320801 | 1000 | 2021-05-10 08:38:13 | 2021-05-10 08:43:13 |
| 310 | 1 | 0.037051 Hrs | 0.037051115185022354 | 1000 | 2021-05-10 08:43:13 | 2021-05-10 08:48:13 | <--- usage record for backup with offering ID = 1
| 311 | 2 | 0.043229 Hrs | 0.04322861135005951 | 0 | 2021-05-10 08:43:13 | 2021-05-10 08:48:13 | <--- usage record for backup with offering ID = 2
| 318 | 2 | 0.083334 Hrs | 0.08333417028188705 | 1000 | 2021-05-10 08:48:13 | 2021-05-10 08:53:13 |
```
However, when the backup offering is removed from the VM and the VM is then re-assigned to the **SAME** backup offering, the usage gets aggregated, which in my opinion is the way it should behave.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd edited a comment on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd edited a comment on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836266129
We may also need to investigate how usage records are generated for any resource where the usage is less than the aggregation interval. In general I agree a new backup record can be created for every offering change. However the usage records shouldn't club the usage but show individual record per offering (say someone switched from platinum to gold offering and we didn't bill/record usage for when they were on platinum offering).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-881798444
@Pearl1594 @rhtyd @olivierlemasle tentatively assigning it to 4.16 milestone
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836538996
Let's discuss on the mailing list, sounds like a new behaviour/change that may require consensus on dev/users ML.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-831094528
Tentatively trained to 4.15.1
cc @shwstppr @davidjumani @Spaceman1984 @Pearl1594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd edited a comment on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd edited a comment on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836538996
Let's discuss on the mailing list, sounds like a new behaviour/change that may require consensus on dev/users ML. cc @Pearl1594 pl start a discussion thread, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-838371947
Alright un-assigning the milestone, we can look into 4.15.2.0 or 4.16.0.0 based on when the PR is ready for review/testing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd edited a comment on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd edited a comment on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-831094528
Tentatively assigned to 4.15.1
cc @shwstppr @davidjumani @Spaceman1984 @Pearl1594
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-838361355
I agree, targeting it for the next milestone would be better, since we are looking at also changing the way data is to be stored for backup usage records.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] nvazquez commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
nvazquez commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-899188130
As @Pearl1594 advised on https://github.com/apache/cloudstack/pull/5005#issuecomment-896470785 moving forward with PR #5259 to fix this issue
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-838340131
This may require a more broad fix, should we move it to the next milestone @olivierlemasle @Pearl1594 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] Pearl1594 commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
Pearl1594 commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-833278819
@olivierlemasle Over the time-period of 15:00:00 - 15:59:59, since there was a backup taken before removing the offering from the VM and reassigning another backup offering, as per my understanding, in that one hour time frame, the cumulative size (backup size taken between 3PM to 3:15 and after the disassociation and re-assign of another backup offering) would be the usage. Subsequent backup usage will not consider the cumulative usage. cc @rhtyd Could you please confirm if this is the expected behavior?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836517717
Yes, I also think we should aggregate the usages over timerange when the VM is re-assigned to the same backup offering, **IF** we sum durations, not sizes (#5005). :+1:
So #5005 (not reviewed yet) should indeed fix the issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-912340879
Fixed by linked PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4982: Incorrect backup usage when VM is removed then re-assign to backup offering
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4982:
URL: https://github.com/apache/cloudstack/issues/4982#issuecomment-836266129
We may also need to investigate how usage records are generated for any resource where the usage is less than the aggregation interval. In general I agree a new backup record can be created for every offering change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org