You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by "mwa-sudo (via GitHub)" <gi...@apache.org> on 2026/06/11 11:14:13 UTC
[I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
mwa-sudo opened a new issue, #13399:
URL: https://github.com/apache/cloudstack/issues/13399
### problem
We are experiencing what appears to be the same issue described in #13112, however in our environment it affects `usage_type = 1` (default VM usage) instead of the usage type reported in the original issue.
The Usage Server repeatedly reprocesses historical usage records from a fixed timestamp and continuously inserts duplicate records into the `cloud_usage` table.
### **observed behaviour**
The Usage Server repeatedly processes the same historical records and generates duplicate rows in `cloud_usage`.
* start date repeatedly reprocessed: `2026-05-29T22:00:00+0200`
### **relation to #13112**
This issue appears very similar to #13112.
Differences:
* #13112 reported the problem for usage_type = `13`
* We observe the same behaviour for usage_type = `1`
* The issue was reported as fixed in a later CloudStack release by the original Poster, however we are running `4.22.1.0` which is newer than the reported version.
Because the symptoms are effectively identical, it indicates that the issue still persists in newer versions.
### **evidence**
Relevant Usage Server logs:
During the period where duplicate usage records are generated, the Usage Server repeatedly processes historical time ranges and emits SQL integrity constraint violations.
A log snippet is attached. VM and account names have been redacted from the logs for data protection purposes.
[_usage.2026-05-28.log](https://github.com/user-attachments/files/28835661/_usage.2026-05-28.log)
A representative example is shown below:
`2026-05-29 23:15:00,005 INFO Parsing usage records between [2026-05-29T20:00:00+0000] and [2026-05-29T20:59:59+0000]
ERROR: Duplicate entry '570-2026-05-29 20:50:40' for key 'id' java.sql.SQLIntegrityConstraintViolationException
WARN: Failed to create usage event id: 1703940 type: VOLUME.CREATE due to Entity already exists`
The exception indicates that the Usage Server is attempting to insert records that already exist. This appears to coincide with the repeated processing of historical usage intervals and the continuous growth of duplicate usage records described above.
To quantify the issue, we wrote a Python script that hashes usage records and counts identical entries for a single virtual machine grouped by hour.
The expected result is exactly one record per hourly interval (`count = 1`).
This expectation is met until `2026-05-29 22:00`, after which duplicate records begin appearing. The duplication follows a distinct pattern:
* `2026-05-29 22:00` → 204 identical records
* `2026-05-29 23:00` → 203 identical records
* `2026-05-30 00:00` → 202 identical records
* `2026-05-30 01:00` → 201 identical records
The pattern continues with each subsequent hourly bucket containing one fewer duplicate than the previous bucket.
This suggests that historical usage intervals are being repeatedly regenerated or reprocessed by the Usage Server. While the total number of records continues to increase over time, the number of duplicates associated with each successive hourly interval decreases by one.
Example output for a single VM:
`{`\
` "2026-05-29": {`\
` "21": 1,`\
` "22": 204,`\
` "23": 203`\
` },`\
` "2026-05-30": {`\
` "00": 202,`\
` "01": 201,`\
` "02": 200,`\
` "03": 199`\
` }`\
`}`
### versions
* Hypervisor: `KVM`
* Database: `10.11.14-MariaDB`
* Apache CloudStack `4.22.1.0`
* Server: `-0ubuntu0.24.04.1-log Ubuntu 24.04`
### The steps to reproduce the bug
The exact trigger is currently unknown. In our environment, the issue can be observed with the following setup:
1. Enable the Usage Server in Apache CloudStack.
2. Configure:
- `usage.stats.job.aggregation.range = 60`
- `usage.stats.job.exec.time = 00:15`
3. Allow the Usage Server to run normally for an extended period.
4. Observe the `cloud_usage` table and Usage Server logs.
5. Duplicate usage records begin appearing and continue to accumulate over time.
### What to do about it?
One observation that might be relevant, is a start-timestamp on 2026-05-29 at 20:00:14 (UTC). This is the only differing usage record. All other duplicates are exactly the same (within VM, day and hour) and their start-timestamps are all for second "00". Without knowing the actual contexts, it does look as if this one existing usage record for second "14" kind of congests the system now. Meaning the server did create an entry for second "14". At the same time it tried to create an entry for second "00". This failed due to the constraint-violation by having the same ID as the entry for second "14". This is observable in the logs.
Starting the following hour the system is indeed creating entries for all subsequent usage records including the one for the timestamp in question. But somehow maybe because of this one "failed" entry, the system does not recognize all subsequently created entries as valid/processed and recreates all of them again every hour.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5325480665
@DaanHoogland both raised:
- #13905
- #13906
I couldn't attach them as sub-issues, that needs write/triage on the repo which I don't have, so they're standalone for now. Happy for you or anyone with permissions to link them under this one.
Schema PR to follow, option 1 as discussed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5193866741
@DaanHoogland we are getting closer. It looks like it is always a VOLUME.CREATE event which causes usage to fail.
grep -nE "Duplicate entry|IntegrityConstraint|Failed to create usage|Exception" /var/log/cloudstack/usage/usage.log
```
195366:2026-08-05 16:15:10,986 ERROR [usage.dao.UsageVolumeDaoImpl_EnhancerByCloudStack_3338cf7d] (Usage-Job-1:[]) (logid:) DB Exception on: HikariProxyPreparedStatement@410370443 wrapping com.mysql.cj.jdbc.ServerPreparedStatement[56]: INSERT INTO usage_volume (usage_volume.zone_id, usage_volume.account_id, usage_volume.domain_id, usage_volume.volume_id, usage_volume.disk_offering_id, usage_volume.template_id, usage_volume.size, usage_volume.vm_id, usage_volume.created, usage_volume.deleted) VALUES (1, 2, 1, 2007, 6, 316, 107374182400, 1874, '2026-08-05 09:40:05', NULL) java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '2007-2026-08-05 09:40:05' for key 'usage_volume.id'
195367: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
195368: at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
195401:2026-08-05 16:15:10,986 DEBUG [db.Transaction.Transaction] (Usage-Job-1:[]) (logid:) Rolling back the transaction: Time = 2242 Name = -UsageManagerImpl.parse:693-UsageManagerImpl.runInContextInternal:427-UsageManagerImpl$1.runInContext:361-ManagedContextRunnable$1.run:49-DefaultManagedContext$1.call:56-DefaultManagedContext.callWithContext:103-DefaultManagedContext.runWithContext:53-ManagedContextRunnable.run:46-UsageManagerImpl.run:366-Executors$RunnableAdapter.call:539-FutureTask.runAndReset:305-ScheduledThreadPoolExecutor$ScheduledFutureTask.run:305; called by -TransactionLegacy.rollback:905-TransactionLegacy.removeUpTo:848-TransactionLegacy.close:672-TransactionContextBuilder.interceptException:63-ComponentInstantiationPostProcessor$InterceptorDispatcher.intercept:131-UsageManagerImpl.createVolumeHelperEvent:1483-UsageManagerImpl.createHelperRecord:959-UsageManagerImpl.parse:711-UsageManagerImpl.runInContextInternal:427-UsageManagerImpl$1.runInContext:361-ManagedContex
tRunnable$1.run:49-DefaultManagedContext$1.call:56
195402:2026-08-05 16:15:10,991 WARN [cloud.usage.UsageManagerImpl_EnhancerByCloudStack_3727543e] (Usage-Job-1:[]) (logid:) Failed to create usage event id: 826827 type: VOLUME.CREATE due to Entity already exists javax.persistence.EntityExistsException: Entity already exists
195403: at com.cloud.utils.db.GenericDaoBase.handleEntityExistsException(GenericDaoBase.java:892)
195427:Caused by: java.sql.SQLIntegrityConstraintViolationException: Duplicate entry '2007-2026-08-05 09:40:05' for key 'usage_volume.id'
195428: at com.mysql.cj.jdbc.exceptions.SQLError.createSQLException(SQLError.java:118)
195429: at com.mysql.cj.jdbc.exceptions.SQLExceptionsMapping.translateException(SQLExceptionsMapping.java:122)
```
SELECT id, type, resource_id, created, processed FROM usage_event WHERE processed = 0
```
+--------+-------------------------+-------------+---------------------+-----------+
| id | type | resource_id | created | processed |
+--------+-------------------------+-------------+---------------------+-----------+
| 826822 | BACKUP.USAGE.METRIC | 1693 | 2026-08-05 09:39:00 | 0 |
| 826823 | BACKUP.USAGE.METRIC | 1182 | 2026-08-05 09:39:00 | 0 |
| 826824 | BACKUP.USAGE.METRIC | 963 | 2026-08-05 09:39:00 | 0 |
| 826825 | BACKUP.USAGE.METRIC | 1485 | 2026-08-05 09:39:00 | 0 |
| 826826 | BACKUP.USAGE.METRIC | 474 | 2026-08-05 09:39:00 | 0 |
| 826827 | VOLUME.CREATE | 2007 | 2026-08-05 09:40:05 | 0 |
+--------+-------------------------+-------------+---------------------+-----------+
```
SELECT id, type, resource_id, created, processed FROM cloud_usage.usage_event WHERE resource_id = 2007 ORDER BY id;
```
+--------+---------------------+-------------+---------------------+-----------+
| id | type | resource_id | created | processed |
+--------+---------------------+-------------+---------------------+-----------+
| 10042 | SNAPSHOT.ON_PRIMARY | 2007 | 2024-10-01 00:41:56 | 1 |
| 10085 | SNAPSHOT.CREATE | 2007 | 2024-10-01 01:41:13 | 1 |
| 10698 | SNAPSHOT.DELETE | 2007 | 2024-10-08 01:57:17 | 1 |
| 66660 | NET.RULEADD | 2007 | 2025-09-29 10:31:48 | 1 |
| 67980 | NET.RULEDELETE | 2007 | 2025-10-06 09:43:21 | 1 |
| 826827 | VOLUME.CREATE | 2007 | 2026-08-05 09:40:05 | 0 |
| 826834 | VOLUME.DELETE | 2007 | 2026-08-05 09:43:04 | 1 |
+--------+---------------------+-------------+---------------------+-----------+
```
It looks like usage cron is doing the INSERT twice:
grep -i "usage_volume" /var/lib/mysql/mgmt-server.log | grep 2007
```
2026-08-05T15:15:10.854502Z 98385 Execute SELECT usage_volume.id, usage_volume.zone_id, usage_volume.account_id, usage_volume.domain_id, usage_volume.volume_id, usage_volume.disk_offering_id, usage_volume.template_id, usage_volume.size, usage_volume.vm_id, usage_volume.created, usage_volume.deleted FROM usage_volume WHERE usage_volume.account_id = 2 AND usage_volume.volume_id = 2007 AND usage_volume.deleted IS NULL
2026-08-05T15:15:10.856192Z 98385 Execute INSERT INTO usage_volume (usage_volume.zone_id, usage_volume.account_id, usage_volume.domain_id, usage_volume.volume_id, usage_volume.disk_offering_id, usage_volume.template_id, usage_volume.size, usage_volume.vm_id, usage_volume.created, usage_volume.deleted) VALUES (1, 2, 1, 2007, 6, 316, 107374182400, NULL, '2026-08-05 09:40:05', NULL)
2026-08-05T15:15:10.860104Z 98385 Execute INSERT INTO usage_volume (usage_volume.zone_id, usage_volume.account_id, usage_volume.domain_id, usage_volume.volume_id, usage_volume.disk_offering_id, usage_volume.template_id, usage_volume.size, usage_volume.vm_id, usage_volume.created, usage_volume.deleted) VALUES (1, 2, 1, 2007, 6, 316, 107374182400, 1874, '2026-08-05 09:40:05', NULL)
```
Claude tells me that this is a regression from 4.22.0, because the vm_id was added to a volume usage record.
There is a broken case EventTypes.EVENT_VOLUME_CREATE in usage/src/main/java/com/cloud/usage/UsageManagerImpl.java.
The `listByVolumeId(...) → deleteExistingVolumeUsage(...)` check runs before both inserts and only finds rows that have already been committed. However, the two problematic inserts are produced by the same code block immediately one after another within the same transaction — and the safeguard doesn't protect against that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5201977995
> I have no idea about the usage server code, nor how it interacts with the rest of cloudstack. But the Linstor plugin didn't change anything about volume create or attaching and if you say this is a regression, I guess it is some other part that tries a duplicate insert for some reason.
Thx for answering! usage was changed and vm_id was added to a volume usage record in 4.22.1. So my best guess was that linstor plugin is creating the volume first without any attaching, which will be an entry in usage and after that it will attach the volume to the vm which will create the second usage entry.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5128502196
> [@mwa-sudo](https://github.com/mwa-sudo) , is [@sbrueseke](https://github.com/sbrueseke) ’s workaround useful for you?
@DaanHoogland it looks like this workaround is only a temporary fix. After some days of normal usage running, the issue is back with another start_date hanging. We restarted our investigation. This happened to both ACS installations we fixed with this workaround.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5326784249
PR is up: #13909
Adds the two `cloud_usage`-schema procedures and has the migration call those, per your last comment @DaanHoogland.
One thing I flagged in the PR description rather than deciding myself: `CONTRIBUTING.md` says bug fixes should target a release branch, which would be `4.22` — but `4.22` has no 4.22.1 → 4.22.2 upgrade path yet (its newest is `Upgrade42200to42210.java`), whereas `main` already has `Upgrade42210to42300.java`. Creating a 4.22.2 path seemed like a release-management call rather than mine, so I targeted `main`.
If it ought to live in `4.22` instead, a cherry-pick or retarget by someone familiar with the release process would be safer than me creating a 4.22.2 upgrade path from scratch, happy to follow guidance if that's the route.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5128605468
What we see after the second time using the workaround is the following:
```
SELECT * FROM ( SELECT * FROM cloud_usage.usage_job ORDER BY id DESC LIMIT 10 ) sub ORDER BY id ASC;
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat |
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 52470 | pc-cm-1.fra1.proio.local/127.0.1.1 | 916185 | 0 | 0 | 1784797248000 | 1785376799999 | 5744828 | 2026-07-23 09:00:48 | 2026-07-30 01:59:59 | 1 | 2026-07-30 03:42:10 |
| 52471 | pc-cm-1.fra1.proio.local/127.0.1.1 | 916185 | 0 | 0 | 1784797248000 | 1785380399999 | 5772631 | 2026-07-23 09:00:48 | 2026-07-30 02:59:59 | 1 | 2026-07-30 05:19:10 |
| 52472 | pc-cm-1.fra1.proio.local/127.0.1.1 | 916185 | 0 | 0 | 1784797248000 | 1785387599999 | 5861548 | 2026-07-23 09:00:48 | 2026-07-30 04:59:59 | 1 | 2026-07-30 06:56:10 |
| 52473 | pc-cm-1.fra1.proio.local/127.0.1.1 | 916185 | 0 | 0 | 1784797248000 | 1785391199999 | 5861860 | 2026-07-23 09:00:48 | 2026-07-30 05:59:59 | 1 | 2026-07-30 08:34:10 |
| 52475 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22701 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:39:36 |
| 52476 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22347 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:40:36 |
| 52477 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22082 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:41:36 |
| 52478 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22321 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:42:36 |
| 52479 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22058 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:43:36 |
| 52480 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 0 | 0 | 0 | NULL | NULL | NULL | 2026-07-30 08:43:58 |
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
10 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5325689549
@Alpha162 reading back your earlier message, I think you may need the combination of options 1 and 2 as we are changing keys in the cloud_usage scheme.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5367753667
> 💡 Notes and suggestions
> The issue thread already contains a thorough root-cause analysis (constraint violation on `cloud_usage`/`usage_event` due to a non-idempotent volume-create usage event) and a fix is in review via PR [#13909](https://github.com/apache/cloudstack/pull/13909). Maintainers should track this issue against that PR and close both [#13905](https://github.com/apache/cloudstack/issues/13905)/[#13906](https://github.com/apache/cloudstack/issues/13906) as resolved once [#13909](https://github.com/apache/cloudstack/pull/13909) merges. No further action needed from triage beyond linking.
these remarks by the bot are misleading, the sub-issues are not solved by the index adjustment. @waterWang’s PRs are, but need back porting to older LTS branches.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-4679939420
Thanks for opening your first issue here! Be sure to follow the issue template!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5132874263
> anything you did between 2026-07-30 08:59:36 and 2026-07-30 09:02:36, [@sbrueseke](https://github.com/sbrueseke) ? also, it suddenly processes a 3 hour window instead of a 2 hour one??? and for some reason that is successful?
@DaanHoogland no, not that we are aware of.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5296293522
**Same issue on 4.22.1.0. I think we have the root cause, with heavy assistance from Claude — there are two defects compounding, and the trigger is unconditional.**
### TL;DR
`createVolumeHelperEvent()` issues **two** inserts into `usage_volume` for a single `VOLUME.CREATE`, with identical `(volume_id, created)`. The table's unique key is `(volume_id, created)`. The second insert can therefore **never** succeed. `GenericDaoBase.persist()` then leaves the caller's transaction unbalanced on the exception path, so the batch's `processed` flags are silently discarded — which pins the aggregation start date forever via the rewind in `parse()`.
The `catch (EntityExistsException)` in `createHelperRecord()` is intended to log-and-continue, but by the time it runs the enclosing transaction is already unrecoverable, so continuing achieves nothing.
### Defect 1 — the duplicate insert is guaranteed by construction
`UsageManagerImpl.createVolumeHelperEvent()`, `EVENT_VOLUME_CREATE` branch (4.22.1.0, ~L1478-1486):
volumeVO = new UsageVolumeVO(volId, ..., null, event.getSize(), event.getCreateDate(), null);
_usageVolumeDao.persist(volumeVO); // vm_id = NULL
if (event.getVmId() != null) {
volumeVO = new UsageVolumeVO(volId, ..., event.getVmId(), event.getSize(), event.getCreateDate(), null);
_usageVolumeDao.persist(volumeVO); // vm_id set, SAME volId + createDate
}
Against the current schema:
UNIQUE KEY `id` (`volume_id`,`created`)
`vm_id` and the second `persist()` were added in 4.22.1; the unique key was not widened to include it. Insert #1 succeeds, insert #2 always violates the constraint.
**Trigger: any `VOLUME.CREATE` where `vm_id` is set — i.e. every ROOT disk, i.e. every VM creation.** We do **not** run Linstor, so this is not plugin-specific.
### Defect 2 — `persist()` leaves the caller's transaction unbalanced
`GenericDaoBase.persist()` (~L1625-1682):
final TransactionLegacy txn = TransactionLegacy.currentTxn(); // the CALLER's transaction
try {
txn.start(); // pushes a START_TXN nesting level
...
pstmt.executeUpdate(); // throws here
...
txn.commit(); // never reached
} catch (final SQLException e) {
logger.error("DB Exception on: " + pstmt, e);
handleEntityExistsException(e); // throws EntityExistsException
throw new CloudRuntimeException("Unable to persist on DB, due to: " + e.getLocalizedMessage());
}
// no finally — the pushed nesting level is never released
`persist()` does not own a transaction; it joins the caller's (`usageTxn`, opened in `parse()`). With no `finally`, the nesting level pushed by `txn.start()` is never popped when `executeUpdate()` throws. `parse()`'s `usageTxn.commit()` then finds the transaction unbalanced, and `TransactionLegacy.commit()` (L723-727) no-ops:
public boolean commit() {
if (!_txn) {
LOGGER.warn("txn: Commit called when it is not a transaction: " + buildName());
return false;
}
Everything in that transaction is discarded — including the `event.setProcessed(true)` updates for the whole batch, and the *successful* insert #1.
### Defect 3 (arguably by design) — the rewind has no lower bound
`parse()` (~L697-703):
Date oldestEventDate = events.get(0).getCreateDate();
if (oldestEventDate.getTime() < startDateMillis) {
startDateMillis = oldestEventDate.getTime();
startDate = new Date(startDateMillis);
}
`events` is `listLatestEvents()` — `WHERE processed = 0 AND created <= ? ORDER BY createDate ASC`. Because defect 2 means those flags never persist, and this rewind only ever moves *backwards*, a single permanently-unprocessed event pins the aggregation start indefinitely. The checkpoint lookup itself (`getLastJobSuccessDateMillis()`) is correct — its result is simply overwritten.
### Evidence from our environment
Unusually clean, because this happened on a brand-new cluster: the six stuck events are **ids 1-6, the first six events the deployment ever emitted**.
SELECT COUNT(*), SUM(processed=0), MIN(CASE WHEN processed=0 THEN created END)
FROM cloud_usage.usage_event;
-- 319 total, 6 unprocessed, oldest 2026-06-04 16:50:51
SELECT COUNT(*) AS successes, MAX(start_millis) FROM cloud_usage.usage_job WHERE success = 1;
-- 1678 successes, MAX(start_millis) = 1780591851000 = 2026-06-04 16:50:51
1,678 successful jobs, and `MAX(start_millis)` never once advanced past the oldest unprocessed event. `exec_time` on recent jobs is **2,555,055 ms (42.6 min)** of every hour, growing by 24 hourly periods per day.
**The insert rolls back — confirmed:**
SELECT * FROM cloud_usage.usage_volume WHERE volume_id = <vol>; -- Empty set
CHECK TABLE cloud_usage.usage_volume; -- OK
The colliding row does not exist at rest and the index is sound. `usage_volume` holds 33 rows with `AUTO_INCREMENT=3481` — the gap is ~1,670 failed inserts consuming IDs, one per hourly run since June.
**Log timing places the failure at the batch commit, not the final one:**
00:00:24,103 INFO Parsing usage records between [...23:00:00] and [...23:59:59]
00:00:24,521 ERROR DB Exception ... Duplicate entry '<vol>-2026-06-04 19:45:04' for key 'usage_volume.id'
00:00:24,523 WARN Failed to create usage event id: 6 type: VOLUME.CREATE due to Entity already exists
00:00:24,651 WARN txn: Commit called when it is not a transaction:
00:42:54,429 INFO usage job complete
The commit warning fires 128 ms after the constraint violation and **42 minutes before the job completes**, placing it at the `usageTxn.commit()` following the event loop rather than the job's final commit.
**Duplication is exactly quantifiable.** For any single hour, every duplicate count is an exact multiple of the number of runs that have covered it. Example for one hour that had been re-aggregated 61 times:
SELECT account_id, usage_type, start_date, end_date, COUNT(*) AS copies
FROM cloud_usage.cloud_usage
WHERE start_date >= '<hour>' AND start_date < '<hour+1>'
GROUP BY account_id, usage_type, start_date, end_date ORDER BY copies DESC;
-- 793 (=61x13), 732 (=61x12), 244 (=61x4), 183 (=61x3), 122 (=61x2), 61 (=61x1)
i.e. `copies = resource_instances x re-aggregations`. `cloud_usage` currently holds **54,460,998 rows / 14.08 GB** where roughly 150 K rows are genuine, growing ~3.2 M rows (~830 MB) per day and accelerating.
### Why the existing workaround only lasts a few days
Re: the report above that clearing the stuck state works and then regresses "after some days" — that isn't time-based. Because defect 1 fires on **every** `VOLUME.CREATE` with `vm_id` set, the next VM created re-wedges it deterministically. On a cluster with regular instance churn it will recur almost immediately.
### Suggested fixes
1. **`GenericDaoBase.persist()` needs a `finally` that releases the nesting level.** This is the highest-value fix — it repairs a whole class of bug, since any caught `EntityExistsException` anywhere in the codebase currently leaves the caller's transaction silently unrecoverable while the caller believes it recovered.
2. **Reconcile `createVolumeHelperEvent()` with the schema.** Either widen the unique key to `(volume_id, created, vm_id)`, or don't emit two rows for the same `(volume_id, created)`. **Open question for maintainers:** does `VolumeUsageParser` aggregate both rows? If so, widening the key would swap a wedged usage server for silently double-counted volume usage, which seems worse. Someone who knows the intent behind adding `vm_id` should decide this.
3. **Bound the rewind in `parse()`** so a permanently-unprocessable event cannot pin aggregation indefinitely — defence in depth, independent of 1 and 2.
### Operator workaround (mitigation, not a fix)
UPDATE cloud_usage.usage_event SET processed = 1 WHERE id IN (<stuck ids>);
Do **not** delete the rows — `getMostRecentEventId()` is `ORDER BY id DESC LIMIT 1` over the whole table, so emptying it returns 0 and triggers `COPY_ALL_EVENTS`, re-copying every event with `processed = 0`.
Worth monitoring the age of the oldest unprocessed event, since this will recur:
SELECT COALESCE(TIMESTAMPDIFF(MINUTE, MIN(created), NOW()), 0)
FROM cloud_usage.usage_event WHERE processed = 0;
### Environment
- CloudStack **4.22.1.0** (EL9 packages, unmodified upstream build — stack-trace line numbers match the `4.22.1.0` tag exactly across `UsageManagerImpl`, `GenericDaoBase` and `TransactionLegacy`)
- MySQL 8.x / InnoDB
- `usage.stats.job.aggregation.range = 60`, `usage.stats.job.exec.time = 00:15`, both timezones UTC
- `usage.sanity.check.interval` unset
- No Linstor
---
---
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5047530258
@mwa-sudo , is @sbrueseke ’s workaround useful for you?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "mwa-sudo (via GitHub)" <gi...@apache.org>.
mwa-sudo commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5128591318
> @mwa-sudo , is @sbrueseke ’s workaround useful for you?
@DaanHoogland sorry, I can't tell. Since we couldn't rely on the usage data for our billing, we ended up disabling the Usage Server entirely. Instead, we now use the API and custom scripts to fetch the data we need. We would still be very interested in a reliable fix.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5356775126
## 🎯 Triage report
Usage Server repeatedly reprocesses historical usage intervals and inserts duplicate `cloud_usage` rows due to a unique-key/constraint-violation loop, affecting `usage_type=1` (VM usage) on 4.22.1.0. This is a well-documented variant of #13112. Root cause has since been identified in the comment thread (a defect introduced by #11531 around `createVolumeHelperEvent` handling), and a fix PR (#13909, with supporting DB migration issues #13905/#13906) is already up.
### 📊 Assessment
| Dimension | Value | Reasoning |
|---|---|---|
| **Type** | type:bug | Reproducible defect causing data corruption in usage accounting |
| **Component** | component:usage-server | Issue is entirely within Usage Server processing/DB schema |
| **Severity** | Severity:Major | Corrupts billing data and requires disabling the Usage Server as a workaround, but is not a full outage |
| **Labels** | type:bug, component:usage-server, Severity:Major | See above |
| **Coding agent** | Not suitable | Root cause analysis and fix are already in progress by a human contributor with PR #13909 open |
### 🔗 Similar issues
- https://github.com/apache/cloudstack/issues/13112 — related, same duplicate-usage-record symptom for a different `usage_type`, referenced directly in this issue
<details><summary>💡 Notes and suggestions</summary>
The issue thread already contains a thorough root-cause analysis (constraint violation on `cloud_usage`/`usage_event` due to a non-idempotent volume-create usage event) and a fix is in review via PR #13909. Maintainers should track this issue against that PR and close both #13905/#13906 as resolved once #13909 merges. No further action needed from triage beyond linking.
</details>
> Generated by [Daily Issue Triage](https://github.com/apache/cloudstack/actions/runs/32375697891) · sonnet50 224.8K · [◷](https://github.com/search?q=repo%3Aapache%2Fcloudstack+%22gh-aw-workflow-call-id%3A+apache%2Fcloudstack%2Fdaily-issue-triage%22&type=issues)
>
<details>
<summary>Add this agentic workflows to your repo</summary>
To install this agentic workflow, run
```
gh aw add githubnext/agentics/workflows/daily-issue-triage.md@d7c1dc4b72b00607a67caaffdcc216cb64379cf9
```
</details>
<!-- gh-aw-agentic-workflow: Daily Issue Triage, engine: copilot, version: 1.0.52, model: claude-sonnet-5, id: 32375697891, workflow_id: daily-issue-triage, run: https://github.com/apache/cloudstack/actions/runs/32375697891 -->
<!-- gh-aw-workflow-call-id: apache/cloudstack/daily-issue-triage -->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5176070498
@DaanHoogland we run into it again. Here is the output of the database. Andy idea where to look to find the root cause? Something started at 2026-08-03 16:02:43.
`mysql> SELECT id, type, resource_id, created, processed FROM cloud_usage.usage_event WHERE processed = 0;
+--------+---------------------+-------------+---------------------+-----------+
| id | type | resource_id | created | processed |
+--------+---------------------+-------------+---------------------+-----------+
| 817825 | BACKUP.USAGE.METRIC | 1381 | 2026-08-03 16:02:43 | 0 |
| 817826 | BACKUP.USAGE.METRIC | 1258 | 2026-08-03 16:02:43 | 0 |
| 817827 | BACKUP.USAGE.METRIC | 1394 | 2026-08-03 16:02:43 | 0 |
| 817828 | BACKUP.USAGE.METRIC | 1398 | 2026-08-03 16:02:43 | 0 |
| 817829 | BACKUP.USAGE.METRIC | 1532 | 2026-08-03 16:02:43 | 0 |
| 817830 | BACKUP.USAGE.METRIC | 1536 | 2026-08-03 16:02:43 | 0 |
| 817831 | BACKUP.USAGE.METRIC | 1665 | 2026-08-03 16:02:43 | 0 |
| 817832 | BACKUP.USAGE.METRIC | 1411 | 2026-08-03 16:02:43 | 0 |
| 817833 | BACKUP.USAGE.METRIC | 1797 | 2026-08-03 16:02:43 | 0 |
| 817834 | BACKUP.USAGE.METRIC | 1670 | 2026-08-03 16:02:43 | 0 |
| 817835 | BACKUP.USAGE.METRIC | 1556 | 2026-08-03 16:02:43 | 0 |
| 817836 | BACKUP.USAGE.METRIC | 1560 | 2026-08-03 16:02:43 | 0 |
| 817837 | BACKUP.USAGE.METRIC | 1563 | 2026-08-03 16:02:43 | 0 |
| 817838 | BACKUP.USAGE.METRIC | 1693 | 2026-08-03 16:02:43 | 0 |
| 817839 | BACKUP.USAGE.METRIC | 1567 | 2026-08-03 16:02:43 | 0 |
| 817840 | BACKUP.USAGE.METRIC | 1568 | 2026-08-03 16:02:43 | 0 |
| 817841 | BACKUP.USAGE.METRIC | 1569 | 2026-08-03 16:02:43 | 0 |
| 817842 | BACKUP.USAGE.METRIC | 1570 | 2026-08-03 16:02:43 | 0 |
| 817843 | BACKUP.USAGE.METRIC | 1485 | 2026-08-03 16:02:43 | 0 |
| 817844 | BACKUP.USAGE.METRIC | 1381 | 2026-08-03 16:07:44 | 0 |
| 817845 | BACKUP.USAGE.METRIC | 1258 | 2026-08-03 16:07:44 | 0 |
| 817846 | BACKUP.USAGE.METRIC | 1394 | 2026-08-03 16:07:44 | 0 |
| 817847 | BACKUP.USAGE.METRIC | 1398 | 2026-08-03 16:07:44 | 0 |
| 817848 | BACKUP.USAGE.METRIC | 1532 | 2026-08-03 16:07:44 | 0 |
| 817849 | BACKUP.USAGE.METRIC | 1536 | 2026-08-03 16:07:44 | 0 |
| 817850 | BACKUP.USAGE.METRIC | 1665 | 2026-08-03 16:07:44 | 0 |
| 817851 | BACKUP.USAGE.METRIC | 1411 | 2026-08-03 16:07:44 | 0 |
| 817852 | BACKUP.USAGE.METRIC | 1797 | 2026-08-03 16:07:44 | 0 |
| 817853 | BACKUP.USAGE.METRIC | 1670 | 2026-08-03 16:07:44 | 0 |
| 817854 | BACKUP.USAGE.METRIC | 1556 | 2026-08-03 16:07:44 | 0 |
| 817855 | BACKUP.USAGE.METRIC | 1560 | 2026-08-03 16:07:44 | 0 |
| 817856 | BACKUP.USAGE.METRIC | 1563 | 2026-08-03 16:07:44 | 0 |
| 817857 | BACKUP.USAGE.METRIC | 1693 | 2026-08-03 16:07:44 | 0 |
| 817858 | BACKUP.USAGE.METRIC | 1567 | 2026-08-03 16:07:44 | 0 |
| 817859 | BACKUP.USAGE.METRIC | 1568 | 2026-08-03 16:07:44 | 0 |
| 817860 | BACKUP.USAGE.METRIC | 1569 | 2026-08-03 16:07:44 | 0 |
| 817861 | BACKUP.USAGE.METRIC | 1570 | 2026-08-03 16:07:44 | 0 |
| 817862 | BACKUP.USAGE.METRIC | 1485 | 2026-08-03 16:07:44 | 0 |
| 817863 | BACKUP.USAGE.METRIC | 1381 | 2026-08-03 16:12:44 | 0 |
| 817864 | BACKUP.USAGE.METRIC | 1258 | 2026-08-03 16:12:44 | 0 |
| 817865 | BACKUP.USAGE.METRIC | 1394 | 2026-08-03 16:12:44 | 0 |
| 817866 | BACKUP.USAGE.METRIC | 1398 | 2026-08-03 16:12:44 | 0 |
| 817867 | BACKUP.USAGE.METRIC | 1532 | 2026-08-03 16:12:44 | 0 |
| 817868 | BACKUP.USAGE.METRIC | 1536 | 2026-08-03 16:12:44 | 0 |
| 817869 | BACKUP.USAGE.METRIC | 1665 | 2026-08-03 16:12:44 | 0 |
| 817870 | BACKUP.USAGE.METRIC | 1411 | 2026-08-03 16:12:44 | 0 |
| 817871 | BACKUP.USAGE.METRIC | 1797 | 2026-08-03 16:12:44 | 0 |
| 817872 | BACKUP.USAGE.METRIC | 1670 | 2026-08-03 16:12:44 | 0 |
| 817873 | BACKUP.USAGE.METRIC | 1556 | 2026-08-03 16:12:44 | 0 |
| 817874 | BACKUP.USAGE.METRIC | 1560 | 2026-08-03 16:12:44 | 0 |
| 817875 | BACKUP.USAGE.METRIC | 1563 | 2026-08-03 16:12:44 | 0 |
| 817876 | BACKUP.USAGE.METRIC | 1693 | 2026-08-03 16:12:44 | 0 |
| 817877 | BACKUP.USAGE.METRIC | 1567 | 2026-08-03 16:12:44 | 0 |
| 817878 | BACKUP.USAGE.METRIC | 1568 | 2026-08-03 16:12:44 | 0 |
| 817879 | BACKUP.USAGE.METRIC | 1569 | 2026-08-03 16:12:44 | 0 |
| 817880 | BACKUP.USAGE.METRIC | 1570 | 2026-08-03 16:12:44 | 0 |
| 817881 | BACKUP.USAGE.METRIC | 1485 | 2026-08-03 16:12:44 | 0 |
| 817882 | BACKUP.USAGE.METRIC | 1381 | 2026-08-03 16:17:44 | 0 |
| 817883 | BACKUP.USAGE.METRIC | 1258 | 2026-08-03 16:17:44 | 0 |
| 817884 | BACKUP.USAGE.METRIC | 1394 | 2026-08-03 16:17:44 | 0 |
| 817885 | BACKUP.USAGE.METRIC | 1398 | 2026-08-03 16:17:44 | 0 |
| 817886 | BACKUP.USAGE.METRIC | 1532 | 2026-08-03 16:17:44 | 0 |
| 817887 | BACKUP.USAGE.METRIC | 1536 | 2026-08-03 16:17:44 | 0 |
| 817888 | BACKUP.USAGE.METRIC | 1665 | 2026-08-03 16:17:44 | 0 |
| 817889 | BACKUP.USAGE.METRIC | 1411 | 2026-08-03 16:17:44 | 0 |
| 817890 | BACKUP.USAGE.METRIC | 1797 | 2026-08-03 16:17:44 | 0 |
| 817891 | BACKUP.USAGE.METRIC | 1670 | 2026-08-03 16:17:44 | 0 |
| 817892 | BACKUP.USAGE.METRIC | 1556 | 2026-08-03 16:17:44 | 0 |
| 817893 | BACKUP.USAGE.METRIC | 1560 | 2026-08-03 16:17:44 | 0 |
| 817894 | BACKUP.USAGE.METRIC | 1563 | 2026-08-03 16:17:44 | 0 |
| 817895 | BACKUP.USAGE.METRIC | 1693 | 2026-08-03 16:17:44 | 0 |
| 817896 | BACKUP.USAGE.METRIC | 1567 | 2026-08-03 16:17:44 | 0 |
| 817897 | BACKUP.USAGE.METRIC | 1568 | 2026-08-03 16:17:44 | 0 |
| 817898 | BACKUP.USAGE.METRIC | 1569 | 2026-08-03 16:17:44 | 0 |
| 817899 | BACKUP.USAGE.METRIC | 1570 | 2026-08-03 16:17:44 | 0 |
| 817900 | BACKUP.USAGE.METRIC | 1485 | 2026-08-03 16:17:44 | 0 |
| 817901 | NETWORK.CREATE | 411 | 2026-08-03 16:22:08 | 0 |
| 817902 | NET.IPASSIGN | 30 | 2026-08-03 16:22:08 | 0 |
| 817903 | NETWORK.UPDATE | 411 | 2026-08-03 16:22:39 | 0 |
| 817904 | VOLUME.CREATE | 2003 | 2026-08-03 16:22:40 | 0 |
+--------+---------------------+-------------+---------------------+-----------+
`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5296693357
````markdown
**Follow-up — bisected this to a specific PR, and I need to correct one thing I asked above.**
### Introduced by
[#11531](https://github.com/apache/cloudstack/pull/11531) — *"Track volume usage data at a vm granularity as well"* (fixes #10985), merged **2025-11-12**, milestone **4.22.1**.
`schema-42200to42210.sql` lines 22-27 add `vm_id` to `cloud.usage_event`, `cloud_usage.usage_event` and `cloud_usage.usage_volume` via `IDEMPOTENT_ADD_COLUMN`. No index change accompanies it, while `createVolumeHelperEvent()` was changed in the same PR to write two rows sharing `(volume_id, created)`.
**Version bound, verified against the tags:** `4.22.0.0` performs a single `persist()` and its `UsageVolumeVO` has no `vm_id` at all. **4.22.0.0 is unaffected; the defect enters in 4.22.1.0.**
The author's own review comment on `UsageManagerImpl.java:1486` states the requirement precisely:
> "VOLUME_CREATE event will contain the vmId also when it is created during Instance deployment. Need to add a vm specific entry also in that case."
That is exactly the collision case, because **both inserts use `event.getCreateDate()`**. This also refines the trigger I gave above: a volume **attached later** is fine, because the attach path produces a distinct timestamp. It's specifically root disks created *during instance deployment*, where both rows necessarily share `created`. In practice, every VM deployment.
### Correcting my open question
Above I asked whether `VolumeUsageParser` aggregates both rows, and whether widening the unique key would cause double-counting. I can answer that now — it doesn't. #11531 changed the parser's map key to include `vmId`:
```java
- String key = volId + "-" + doId + "-" + size;
+ String key = volId + "-" + doId + "-" + vmId + "-" + size;
```
and `createUsageRecord()` now passes `vmId` into `UsageVO`. The two rows are designed to produce distinct records — cumulative and per-VM — exactly as the PR description sets out. So the missing DDL is simply `UNIQUE KEY (volume_id, created, vm_id)`, and shipping it is aligned with the feature's intent rather than a workaround.
(One nuance for whoever writes it: MySQL treats NULLs in a unique index as distinct, so that key permits multiple `vm_id IS NULL` rows for the same pair. Probably fine, but worth a deliberate decision.)
### The part I think matters most
**#13112 was reported on 4.21.0.0 with `usage_type = 13`.** That predates #11531 by a release, so the volume double-insert cannot have caused it — same symptom, different trigger.
Which means the durable defects are the other two: `GenericDaoBase.persist()` leaving the caller's transaction unbalanced on the exception path, and the unbounded backwards-only rewind in `parse()`. #11531 didn't create the fragility, it just added a very reliable new way to hit it.
So I'd rank the fixes:
1. **`finally` in `GenericDaoBase.persist()`** to release the nesting level. Version-independent, and the reason an isolated constraint violation escalates into permanent aggregation failure instead of being the recoverable log-and-continue that `createHelperRecord()` clearly intends.
2. **Bound the rewind in `parse()`** so a permanently-unprocessable event can't pin aggregation indefinitely.
3. **Ship the missing DDL** for `usage_volume`.
Fixing only (3) closes this particular trigger while leaving the mechanism intact — which would be consistent with #13112 having been closed as fixed and the symptom returning here.
````
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5324819255
@Alpha162 , I actually think your choice is as good as mine in this case. I would emphasise idem-potency as much as possible for any DB action, as in the past upgrades would fail and the only way to roll back or repair was a restore.
Can you create subissues for the other two pieces of work?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5326055328
@DaanHoogland understood, I'll add `usage.idempotent_drop_unique_key.sql` and `usage.idempotent_add_unique_key.sql` under `db/procedures/` as `cloud_usage`-schema copies of the `cloud` versions, and have the migration call those instead. Keeps `cloud_usage` DDL going through `cloud_usage` procedures like the rest of the file. PR shortly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5128832703
After the next scheduled run of usage, success is 1.
```
SELECT * FROM ( SELECT * FROM cloud_usage.usage_job ORDER BY id DESC LIMIT 10 ) sub ORDER BY id ASC;
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat |
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 52488 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22004 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:52:36 |
| 52489 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 21922 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:53:36 |
| 52490 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 21983 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:54:36 |
| 52491 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22038 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:55:36 |
| 52492 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22093 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:56:36 |
| 52493 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22033 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:57:36 |
| 52494 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 21968 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:58:36 |
| 52495 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785398399999 | 22006 | 2026-07-30 06:00:00 | 2026-07-30 07:59:59 | 0 | 2026-07-30 08:59:36 |
| 52496 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 1785391200000 | 1785401999999 | 163559 | 2026-07-30 06:00:00 | 2026-07-30 08:59:59 | 1 | 2026-07-30 09:02:36 |
| 52497 | pc-cm-1.fra1.proio.local/127.0.1.1 | 1817733 | 0 | 0 | 0 | 0 | 0 | NULL | NULL | NULL | 2026-07-30 09:09:36 |
+-------+------------------------------------+---------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
10 rows in set (0.00 sec)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5130904561
anything you did between 2026-07-30 08:59:36 and 2026-07-30 09:02:36, @sbrueseke ?
also, it suddenly processes a 3 hour window instead of a 2 hour one??? and for some reason that is successful?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5325282390
@DaanHoogland thanks, I'll go with option 1 on your idempotency point.
Both `` `cloud`.`IDEMPOTENT_DROP_UNIQUE_KEY` `` and `` `cloud`.`IDEMPOTENT_ADD_UNIQUE_KEY` `` already carry `CONTINUE HANDLER`s (`1091, 1025` and `1061` respectively), so a re-run is a no-op in effect and it needs no new procedures. Option 3 is out for exactly the reason you give, the plain `ALTER TABLE ... ADD UNIQUE KEY` isn't idempotent and would fail on a second pass. It also follows the existing precedent in `schema-42100to42200.sql` (lines 88-89), which drops and re-adds `uc_counter__provider__source__value` the same way.
Worth noting for the rollback concern specifically: widening `(volume_id, created)` to `(volume_id, created, vm_id)` is strictly *less* restrictive, so no existing row can violate the new key. The DDL can't fail on data, only on a key-name mismatch, which is the caveat I flagged above.
I'll target `main` (`schema-42210to42300.sql`). Say the word if you'd like a `4.22` backport too and I'll open a second PR against a new `schema-42210to42220.sql`.
And yes, I'll raise the two subissues:
1. `GenericDaoBase.persist()` leaves the caller's transaction unbalanced when an insert throws
2. `UsageManagerImpl.parse()` rewind to the oldest unprocessed usage event is unbounded
I'll link them back here once created.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "Alpha162 (via GitHub)" <gi...@apache.org>.
Alpha162 commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5317853992
> [@Alpha162](https://github.com/Alpha162) are you creating a PR for this?
@DaanHoogland Yes, happy to, but with one caveat: I'm an operator rather than a CloudStack dev, so I'd want to stick to the schema fix and leave the other two to someone who knows the codebase properly.
One convention question before I open it. To widen `cloud_usage.usage_volume`'s `UNIQUE KEY id` from `(volume_id, created)` to `(volume_id, created, vm_id)`, there seem to be three ways to do it and I'd rather ask than guess:
1. Call the existing `` `cloud`.`IDEMPOTENT_DROP_UNIQUE_KEY` `` / `` `cloud`.`IDEMPOTENT_ADD_UNIQUE_KEY` `` with a schema-qualified `'cloud_usage.usage_volume'`. They build their DDL from `in_table_name` via `CONCAT`, so this should work the same way `'cloud.counter'` does in `schema-42100to42200.sql`.
2. Add `usage.idempotent_drop_unique_key.sql` / `usage.idempotent_add_unique_key.sql` under `db/procedures/`, matching the existing `cloud_usage` helpers, currently that schema only has `IDEMPOTENT_ADD_COLUMN`, `IDEMPOTENT_CHANGE_COLUMN` and `IDEMPOTENT_DROP_INDEX`.
3. Use `` `cloud_usage`.`IDEMPOTENT_DROP_INDEX` `` plus a plain `ALTER TABLE ... ADD UNIQUE KEY`.
Which fits the project's conventions best?
One detail that's relevant to all three: the key is *named* `id` but covers `(volume_id, created)`. That's because `setup/db/create-schema-premium.sql` declares it unnamed on `(id, created)`, so MySQL auto-named it after the first column, and when #5785 introduced auto-increment primary keys and the old `id` became `volume_id`, the index kept its generated name. So any `DROP KEY 'id'` is relying on an auto-generated name rather than a deliberate one. Happy to be told that's fine, just flagging it.
Also happy to be told which branch to target, `main` already has `schema-42210to42300.sql`, but `4.22` has nothing after `schema-42200to42210.sql`.
On the other two, the `finally` in `GenericDaoBase.persist()` and bounding the rewind in `parse()`, I'd rather not be the one touching `GenericDaoBase` given how much depends on it. I should also be straight with you that we're rebuilding the affected environment imminently to an earlier version, so I can't offer it as a test bed. Hopefully the analysis above should be enough to reproduce from scratch.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5031208910
We had the same issue on several stage and prod environments. Her are our findings and solution to it.
You see that start_date of the usage_job is stuck.
`SELECT * FROM ( SELECT * FROM cloud_usage.usage_job ORDER BY id DESC LIMIT 10 ) sub ORDER BY id ASC;
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| id | host | pid | job_type | scheduled | start_millis | end_millis | exec_time | start_date | end_date | success | heartbeat |
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+
| 52246 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784365199999 | 25095596 | 2026-06-17 10:00:55 | 2026-07-18 08:59:59 | 1 | 2026-07-18 16:13:25 |
| 52247 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784390399999 | 25288347 | 2026-06-17 10:00:55 | 2026-07-18 15:59:59 | 1 | 2026-07-18 23:14:25 |
| 52248 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784415599999 | 25566185 | 2026-06-17 10:00:55 | 2026-07-18 22:59:59 | 1 | 2026-07-19 06:21:25 |
| 52249 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784440799999 | 25793951 | 2026-06-17 10:00:55 | 2026-07-19 05:59:59 | 1 | 2026-07-19 13:31:25 |
| 52250 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784465999999 | 26110859 | 2026-06-17 10:00:55 | 2026-07-19 12:59:59 | 1 | 2026-07-19 20:46:25 |
| 52251 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784491199999 | 26317753 | 2026-06-17 10:00:55 | 2026-07-19 19:59:59 | 1 | 2026-07-20 04:04:25 |
| 52252 | mgmt-stage1.proio.local/127.0.1.1 | 1353 | 0 | 0 | 1781690455000 | 1784519999999 | 26624851 | 2026-06-17 10:00:55 | 2026-07-20 03:59:59 | 1 | 2026-07-20 11:28:25 |
| 52256 | mgmt-stage1.proio.local/127.0.1.1 | 916185 | 0 | 0 | 0 | 0 | 0 | NULL | NULL | NULL | 2026-07-20 14:59:10 |
+-------+------------------------------------+--------+----------+-----------+---------------+---------------+-----------+---------------------+---------------------+---------+---------------------+`
The reason it is stuck is that we had unprocessed usage_events.
`SELECT id, type, resource_id, created, processed FROM usage_event WHERE processed = 0;
+--------+---------------------+-------------+---------------------+-----------+
| id | type | resource_id | created | processed |
+--------+---------------------+-------------+---------------------+-----------+
| 481831 | BACKUP.USAGE.METRIC | 40 | 2026-06-17 10:00:55 | 0 |
| 481903 | VOLUME.CREATE | 1897 | 2026-06-17 10:26:15 | 0 |
+--------+---------------------+-------------+---------------------+-----------+`
To get usage back on track and using the correct start_date, we needed to set processed to 1.
`UPDATE usage_event SET processed = 1 WHERE processed = 0`
The next usage run was working as expected.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "sbrueseke (via GitHub)" <gi...@apache.org>.
sbrueseke commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5201576868
@DaanHoogland here is the output
SELECT * FROM (SELECT id, type, resource_id, created, processed FROM usage_event WHERE type = 'VOLUME.CREATE' ORDER BY id DESC LIMIT 10) sub ORDER BY id ASC;
```
+--------+---------------+-------------+---------------------+-----------+
| id | type | resource_id | created | processed |
+--------+---------------+-------------+---------------------+-----------+
| 752179 | VOLUME.CREATE | 1952 | 2026-07-16 14:46:21 | 1 |
| 774500 | VOLUME.CREATE | 1956 | 2026-07-23 09:51:20 | 1 |
| 775401 | VOLUME.CREATE | 1985 | 2026-07-23 15:42:22 | 1 |
| 797749 | VOLUME.CREATE | 1999 | 2026-07-29 11:25:28 | 1 |
| 800985 | VOLUME.CREATE | 2000 | 2026-07-30 07:53:38 | 1 |
| 817904 | VOLUME.CREATE | 2003 | 2026-08-03 16:22:40 | 1 |
| 817927 | VOLUME.CREATE | 2004 | 2026-08-03 16:22:53 | 1 |
| 817931 | VOLUME.CREATE | 2005 | 2026-08-03 16:23:05 | 1 |
| 817935 | VOLUME.CREATE | 2006 | 2026-08-03 16:23:17 | 1 |
| 826827 | VOLUME.CREATE | 2007 | 2026-08-05 09:40:05 | 1 |
+--------+---------------+-------------+---------------------+-----------+
```
I already fixed id 826827 by setting processed to 1 via UPDATE.
I sent you a link to the whole Claude conversation.
@rp- Is it possible that linstor is involved? The root cause is that usage will do 2 INSERT into the db with the same timestamp. The first INSERT has no vm_id and the second has a vm_id. Is linstor creating the volume first and than is attaching it to the vm and doing it so fast, that it will have the exact same timestamp? This could be an explanation for the 2 INSERT.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5200938272
@sbrueseke , can you try
```
SELECT id, type, resource_id, created, processed FROM usage_event WHERE type = VOLUME.CREATE;
```
I would expect there to be another one that had already be processed for a second one to be inserted. Claude is right about the extra fields being added to usage_events, but I am not convince about the processing.
Can you share your prompt?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "rp- (via GitHub)" <gi...@apache.org>.
rp- commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5201808598
I have no idea about the usage server code, nor how it interacts with the rest of cloudstack.
But the Linstor plugin didn't change anything about volume create or attaching and if you say this is a regression, I guess it is some other part that tries a duplicate insert for some reason.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5176911428
@sbrueseke , first thing i’d look for it for any usage_event entries that have the same timestamp or a strange order (i.e. a delete before a create, or a stop before a start)
I see no strangities in the fragment you send, so my best gues is that it would occur just before this.
Also check for any exceptions during the run of the job, usually a duplicate key error is the culprit. We have found (and fixed) occasions where the duplicate key was erroneously disallowed and occurances where the constraint was valid but could have been ignored. These two situations are in addition to the event coming in out of order, so to my knowledge three possible causes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Usage Server repeatedly reprocesses historical usage and creates duplicate cloud_usage records for usage_type=1 [cloudstack]
Posted by "DaanHoogland (via GitHub)" <gi...@apache.org>.
DaanHoogland commented on issue #13399:
URL: https://github.com/apache/cloudstack/issues/13399#issuecomment-5316454307
@Alpha162 are you creating a PR for this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@cloudstack.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org