You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by syed <gi...@git.apache.org> on 2016/06/27 20:17:59 UTC
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
GitHub user syed opened a pull request:
https://github.com/apache/cloudstack/pull/1598
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
With object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/syed/cloudstack vhd-compressed-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1598.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1598
----
commit 841783f3359ebe58683613bb0481c3de38153923
Author: Syed <sy...@gmail.com>
Date: 2016-06-27T20:11:14Z
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by syed <gi...@git.apache.org>.
Github user syed commented on the issue:
https://github.com/apache/cloudstack/pull/1598
@swill if you can merge this before you cut the release, it would be awesome.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
### CI RESULTS
```
Tests Run: 85
Skipped: 0
Failed: 0
Errors: 0
Duration: 4h 07m 10s
```
**Associated Uploads**
**`/tmp/MarvinLogs/DeployDataCenter__Jun_27_2016_23_46_56_POC600:`**
* [dc_entries.obj](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/DeployDataCenter__Jun_27_2016_23_46_56_POC600/dc_entries.obj)
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/DeployDataCenter__Jun_27_2016_23_46_56_POC600/failed_plus_exceptions.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/DeployDataCenter__Jun_27_2016_23_46_56_POC600/runinfo.txt)
**`/tmp/MarvinLogs/test_network_HRWEBW:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_network_HRWEBW/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_network_HRWEBW/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_network_HRWEBW/runinfo.txt)
**`/tmp/MarvinLogs/test_vpc_routers_Z3TJCB:`**
* [failed_plus_exceptions.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_vpc_routers_Z3TJCB/failed_plus_exceptions.txt)
* [results.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_vpc_routers_Z3TJCB/results.txt)
* [runinfo.txt](https://objects-east.cloud.ca/v1/e465abe2f9ae4478b9fff416eab61bd9/PR1598/tmp/MarvinLogs/test_vpc_routers_Z3TJCB/runinfo.txt)
Uploads will be available until `2016-08-28 02:00:00 +0200 CEST`
*Comment created by [`upr comment`](https://github.com/cloudops/upr).*
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by syed <gi...@git.apache.org>.
Github user syed closed the pull request at:
https://github.com/apache/cloudstack/pull/1598
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
This is coming back clean. Since it fixes issues in 4.9, I think it should be merged...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1598#discussion_r68779079
--- Diff: core/src/com/cloud/storage/template/VhdProcessor.java ---
@@ -91,27 +97,58 @@ public long getVirtualSize(File file) throws IOException {
return file.length();
}
- protected long getTemplateVirtualSize(File file) throws IOException {
+ protected long getTemplateVirtualSize(File file) throws IOException {
--- End diff --
Can you fix the method indentation here. This should not be indented that extra tab.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by syed <gi...@git.apache.org>.
Github user syed commented on the issue:
https://github.com/apache/cloudstack/pull/1598
Yes @swill I've tested this on my local setup.
On Mon, Jun 27, 2016 at 5:16 PM, Will Stevens <no...@github.com>
wrote:
> @syed <https://github.com/syed> my understanding is that you have tested
> this fix with swift in our lab. That is correct?
>
> I will get CI run against it to make sure this does not break anything
> outside swift.
>
> \u2014
> You are receiving this because you were mentioned.
> Reply to this email directly, view it on GitHub
> <https://github.com/apache/cloudstack/pull/1598#issuecomment-228878063>,
> or mute the thread
> <https://github.com/notifications/unsubscribe/AADaLukgRTWegA80eHmB9lgg9QMRZxv3ks5qQD2mgaJpZM4I_d12>
> .
>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
Good enough. Thanks... :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
@syed, can you do a once through and clean up some of the white space issues in this diff. I have highlighted a couple. The method indent is the most important, but there are a few other cases where the white space can be cleaned up as well. Thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by syed <gi...@git.apache.org>.
GitHub user syed reopened a pull request:
https://github.com/apache/cloudstack/pull/1598
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
With object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/syed/cloudstack vhd-compressed-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1598.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1598
----
commit b0247b53f99ff97fd05d4d8528115ed6e7d497c0
Author: Syed <sy...@gmail.com>
Date: 2016-06-27T20:11:14Z
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on a diff in the pull request:
https://github.com/apache/cloudstack/pull/1598#discussion_r68779423
--- Diff: core/src/com/cloud/storage/template/VhdProcessor.java ---
@@ -91,27 +97,58 @@ public long getVirtualSize(File file) throws IOException {
return file.length();
}
- protected long getTemplateVirtualSize(File file) throws IOException {
+ protected long getTemplateVirtualSize(File file) throws IOException {
byte[] currentSize = new byte[8];
+ byte[] cookie = new byte[8];
byte[] creatorApp = new byte[4];
- try (FileInputStream strm = new FileInputStream(file)) {
- long skipped = strm.skip(file.length() - vhdFooterSize + vhdFooterCreatorAppOffset);
+
+ BufferedInputStream fileStream = new BufferedInputStream(new FileInputStream(file));
+ InputStream strm = fileStream;
+
+ boolean isCompressed = checkCompressed(file.getAbsolutePath());
+
+ if ( isCompressed ) {
+ try {
+ strm = new CompressorStreamFactory().createCompressorInputStream(fileStream);
+ } catch (CompressorException e) {
+ s_logger.info("error opening compressed VHD file " + file.getName());
+ return file.length();
+ }
+ }
+
+ try {
+
--- End diff --
Can you remove the unneeded line break here since I already have another change for you to make. :) Thx...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by syed <gi...@git.apache.org>.
GitHub user syed reopened a pull request:
https://github.com/apache/cloudstack/pull/1598
[CLOUDSTACK-9423] Add ability to get virtual size of compressed VHDs
With object store like Swift as secondary storage, if a compressed VHD is uploaded as a template, the `VHDProcessor` incorrectly calculates the virutal size leading to the template being useless. This fix tries to guess the virtual size by partially decompressing it and falls back to a sensible default which is the size of the file.
Before the fix: template.properties on Swift
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=2894447637315205059
```
After the fix
```
uniquename=routing-1
filename=routing-1.vhd
size=263417314
virtualsize=3145728000
```
Look at the `virutalsize` in both cases
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/syed/cloudstack vhd-compressed-size
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/cloudstack/pull/1598.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1598
----
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
@syed my understanding is that you have tested this fix with swift in our lab. That is correct?
I will get CI run against it to make sure this does not break anything outside swift.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by syed <gi...@git.apache.org>.
Github user syed commented on the issue:
https://github.com/apache/cloudstack/pull/1598
@swill. I've fixed the whitespaces and rebased everything. Should be good now.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by syed <gi...@git.apache.org>.
Github user syed closed the pull request at:
https://github.com/apache/cloudstack/pull/1598
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack pull request #1598: [CLOUDSTACK-9423] Add ability to get virtual ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/cloudstack/pull/1598
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] cloudstack issue #1598: [CLOUDSTACK-9423] Add ability to get virtual size of...
Posted by swill <gi...@git.apache.org>.
Github user swill commented on the issue:
https://github.com/apache/cloudstack/pull/1598
Thank you. I have CI running and I should have results later tonight.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---