You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cloudstack.apache.org by GitBox <gi...@apache.org> on 2021/04/28 13:14:08 UTC
[GitHub] [cloudstack] olivierlemasle opened a new issue #4965: Backups: Truncated data in DB result in listBackups failure if too many volumes
olivierlemasle opened a new issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965
##### ISSUE TYPE
* Bug Report
##### COMPONENT NAME
~~~
DB, Backup
~~~
##### CLOUDSTACK VERSION
~~~
4.14.1.0
4.15.0.0
~~~
##### CONFIGURATION
`backup.framework.enabled = true`
B&R framework enabled, with either Veeam or dummy provider (issue reproduced with both providers)
##### OS / ENVIRONMENT
OS: CentOS Linux 7
Database: issue reproduced with:
- mariadb-5.5.68-1.el7.x86_64 (Server version: `5.5.68-MariaDB MariaDB Server`)
- MySQL 5.5 (Server version: `5.5.62 MySQL Community Server (GPL)`)
##### SUMMARY
When a VM instance with at least 3 volumes is backed up using CloudStack B&R framework, `listBackups` fail because `backup_volumes` data is truncated in database.
##### STEPS TO REPRODUCE
- Enable B&R (`backup.framework.enabled = true`)
- Deploy a VM with a DATA volume
- Add another DATA volume to the VM, which has now 3 volumes (1 ROOT volume, 2 DATA volumes)
- Assign the VM to a backup offering
- Create a backup of the VM
- Try listing backups for the VM
##### EXPECTED RESULTS
~~~
An API response with the VM backups
~~~
##### ACTUAL RESULTS
<!-- What actually happened? -->
<!-- Paste verbatim command output between quotes below -->
~~~
com.google.gson.stream.MalformedJsonException: Unterminated string near
~~~
or
~~~
java.io.EOFException: End of input
~~~
The first error was seen when `cloud.vm.instance.backup_volumes` was:
```
[{"uuid":"72a056e0-abec-4036-a779-575f645e6e0f","type":"DATADISK","size":10737418240,"path":"1630d4b0ad0a436caf146a7ef0486012"},{"uuid":"254d83d3-b0d7-4aea-bed4-585009e0603b","type":"ROOT","size":21474836480,"path":"ROOT-8207_2"},{"uuid":"7e8da461-b75d-42
```
The second error was seen when `cloud.vm.instance.backup_volumes` was:
```
[{"uuid":"fa90e222-f4c2-4034-8ae7-ca61a3e4c5ac","type":"ROOT","size":8589934592,"path":"fa90e222-f4c2-4034-8ae7-ca61a3e4c5ac"},{"uuid":"ffee8d81-77d4-49f5-ac31-4e57fbdb2e40","type":"DATADISK","size":5368709120,"path":"ffee8d81-77d4-49f5-ac31-4e57fbdb2e40"
```
Both errors are caused by a JSON object representing backup volumes, which is truncated at 256 characters in database.
Please note that `cloud.vm.instance.backup_volumes` has data type TEXT, so it should store up to 65,535 characters...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-829089411
> It's probably silently ignored from mysql side.
Actually, there's no MySQL issue there. CloudStack does this truncation here: https://github.com/apache/cloudstack/blob/1e859a0e2ce8271641a1eaa5665a8297bd119d8c/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1535-L1540
For me, the CloudStack data access layer should rather throw an Exception instead of silently truncating data. However, changing that behaviour now might cause some problematic errors elsewhere...
> are you using/testing Veeam B&R provider in your env?
I'm currently testing the B&R framework (more specifically its usage metering part), with the dummy provider in my lab and with Veeam in a customer lab environment (btw I've just sent you an email on that subject)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-828663074
It's probably silently ignored from mysql side. Thanks for the issue and fix @olivierlemasle - are you using/testing Veeam B&R provider in your env?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-828458692
I did some additional tests:
- I updated manually `cloud.vm.instance.backup_volumes` to set a value with a size > 255 characters, and it succeeded.
- I did the same thing with a simple Java program, using the same mysql / java connector, in the same version that what CloudStack uses (mysql-connector-java-8.0.19.jar). It also succeeded...
So it seems CloudStack itself is doing the truncation. Perhaps in its data access layer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd closed issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
rhtyd closed issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-828636388
Ok, I've found the explicit truncation to the default length [here](https://github.com/apache/cloudstack/blob/1e859a0e2ce8271641a1eaa5665a8297bd119d8c/framework/db/src/main/java/com/cloud/utils/db/GenericDaoBase.java#L1532-L1540).
So the fix should be to pass an explicit `length` [here](https://github.com/apache/cloudstack/blob/1e859a0e2ce8271641a1eaa5665a8297bd119d8c/engine/schema/src/main/java/com/cloud/vm/VMInstanceVO.java#L203).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] rhtyd commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
rhtyd commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-829872498
Fixed in https://github.com/apache/cloudstack/pull/4967
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle edited a comment on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
olivierlemasle edited a comment on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-828458692
I did some additional tests:
- I updated manually `cloud.vm_instance.backup_volumes` to set a value with a size > 255 characters, and it succeeded.
- I did the same thing with a simple Java program, using the same mysql / java connector, in the same version that what CloudStack uses (mysql-connector-java-8.0.19.jar). It also succeeded...
So it seems CloudStack itself is doing the truncation. Perhaps in its data access layer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [cloudstack] olivierlemasle commented on issue #4965: Backups: Truncated data in DB results in listBackups failure if too many volumes
Posted by GitBox <gi...@apache.org>.
olivierlemasle commented on issue #4965:
URL: https://github.com/apache/cloudstack/issues/4965#issuecomment-828652531
Created PR #4967 to fix this issue.
However, I'm wondering in which cases it can be a good thing to truncate silently a String too long to fit. Shouldn't it fail instead?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org