You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@cloudstack.apache.org by Tamas Monos <ta...@veber.co.uk> on 2012/11/14 16:31:25 UTC
Critical - DO NOT use 4.0 storage clean-up after upgrade from 3.0.2
Hi,
I had a successful update from 3.0.2->4.0.
Everything was perfect until I've enabled storage-clean-up in the global options.
At the next first time the storage clean-up thread run it marked all systemVM volumes/templates on the primary storages for removal.
It has literally went through all my system VMs and first deleted their volumes than their templates...
Then the system start to complain about file-not found messages, deploy failed...
I'm trying to recover from this but I cannot re-create the secondary storage VM due to:
2012-11-14 14:17:56,580 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://192.168.1.2/share/watford_cloud
But I end up with:
2012-11-14 14:19:55,241 DEBUG [utils.script.Script] (catalina-exec-6:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2012-11-14 14:19:55,241 INFO [storage.template.DownloadManagerImpl] (catalina-exec-6:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
2012-11-14 14:19:55,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Exit value is 1
2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) ln: creating symbolic link `/var/www/html/copy': No such file or directory
2012-11-14 14:19:55,304 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Error in linking err=ln: creating symbolic link `/var/www/html/copy': No such file or directory
2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) Unable to fully initialize the agent because no StartupCommands are returned
2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) server resources successfully discovered by SecondaryStorage
2012-11-14 14:19:55,305 WARN [cloud.api.ApiDispatcher] (catalina-exec-6:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
This is issue is likely caused by the update procedure due to the systemVM were not using the latest template.
They were not re-deployed just re-started with the new systemVM template and possible some references were not updated in other DB tables so the clean-up script found them as not used and destroyed them all.
I'm lucky enough I always have test environment for everything.
Regards
Tamas Monos DDI +44(0)2034687012
Chief Technical Office +44(0)2034687000
Veber: The Hosting Specialists Fax +44(0)871 522 7057
http://www.veber.co.uk<http://www.veber.co.uk/>
Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
Re: Critical - DO NOT use 4.0 storage clean-up after upgrade from 3.0.2
Posted by Chip Childers <ch...@sungard.com>.
Hi Tamas,
Can you open a bug for this?
On Wed, Nov 14, 2012 at 10:31 AM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I had a successful update from 3.0.2->4.0.
> Everything was perfect until I've enabled storage-clean-up in the global options.
>
> At the next first time the storage clean-up thread run it marked all systemVM volumes/templates on the primary storages for removal.
> It has literally went through all my system VMs and first deleted their volumes than their templates...
> Then the system start to complain about file-not found messages, deploy failed...
>
> I'm trying to recover from this but I cannot re-create the secondary storage VM due to:
> 2012-11-14 14:17:56,580 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://192.168.1.2/share/watford_cloud
>
> But I end up with:
> 2012-11-14 14:19:55,241 DEBUG [utils.script.Script] (catalina-exec-6:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2012-11-14 14:19:55,241 INFO [storage.template.DownloadManagerImpl] (catalina-exec-6:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2012-11-14 14:19:55,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Exit value is 1
> 2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) ln: creating symbolic link `/var/www/html/copy': No such file or directory
> 2012-11-14 14:19:55,304 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Error in linking err=ln: creating symbolic link `/var/www/html/copy': No such file or directory
> 2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) Unable to fully initialize the agent because no StartupCommands are returned
> 2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) server resources successfully discovered by SecondaryStorage
> 2012-11-14 14:19:55,305 WARN [cloud.api.ApiDispatcher] (catalina-exec-6:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
>
> This is issue is likely caused by the update procedure due to the systemVM were not using the latest template.
> They were not re-deployed just re-started with the new systemVM template and possible some references were not updated in other DB tables so the clean-up script found them as not used and destroyed them all.
> I'm lucky enough I always have test environment for everything.
>
> Regards
>
> Tamas Monos DDI +44(0)2034687012
> Chief Technical Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522 7057
> http://www.veber.co.uk<http://www.veber.co.uk/>
>
> Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
> Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
>
Re: Critical - DO NOT use 4.0 storage clean-up after upgrade from 3.0.2
Posted by Chip Childers <ch...@sungard.com>.
Hi Tamas,
Can you open a bug for this?
On Wed, Nov 14, 2012 at 10:31 AM, Tamas Monos <ta...@veber.co.uk> wrote:
> Hi,
>
> I had a successful update from 3.0.2->4.0.
> Everything was perfect until I've enabled storage-clean-up in the global options.
>
> At the next first time the storage clean-up thread run it marked all systemVM volumes/templates on the primary storages for removal.
> It has literally went through all my system VMs and first deleted their volumes than their templates...
> Then the system start to complain about file-not found messages, deploy failed...
>
> I'm trying to recover from this but I cannot re-create the secondary storage VM due to:
> 2012-11-14 14:17:56,580 DEBUG [cloud.server.StatsCollector] (StatsCollector-1:null) There is no secondary storage VM for secondary storage host nfs://192.168.1.2/share/watford_cloud
>
> But I end up with:
> 2012-11-14 14:19:55,241 DEBUG [utils.script.Script] (catalina-exec-6:null) Looking for createvolume.sh in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2012-11-14 14:19:55,241 INFO [storage.template.DownloadManagerImpl] (catalina-exec-6:null) createvolume.sh found in /usr/lib64/cloud/common/scripts/storage/secondary/createvolume.sh
> 2012-11-14 14:19:55,243 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Executing: /bin/bash -c ln -sf /mnt/SecStorage /var/www/html/copy
> 2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Exit value is 1
> 2012-11-14 14:19:55,304 DEBUG [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) ln: creating symbolic link `/var/www/html/copy': No such file or directory
> 2012-11-14 14:19:55,304 WARN [storage.resource.NfsSecondaryStorageResource] (catalina-exec-6:null) Error in linking err=ln: creating symbolic link `/var/www/html/copy': No such file or directory
> 2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) Unable to fully initialize the agent because no StartupCommands are returned
> 2012-11-14 14:19:55,305 INFO [cloud.resource.ResourceManagerImpl] (catalina-exec-6:null) server resources successfully discovered by SecondaryStorage
> 2012-11-14 14:19:55,305 WARN [cloud.api.ApiDispatcher] (catalina-exec-6:null) class com.cloud.api.ServerApiException : Failed to add secondary storage
>
> This is issue is likely caused by the update procedure due to the systemVM were not using the latest template.
> They were not re-deployed just re-started with the new systemVM template and possible some references were not updated in other DB tables so the clean-up script found them as not used and destroyed them all.
> I'm lucky enough I always have test environment for everything.
>
> Regards
>
> Tamas Monos DDI +44(0)2034687012
> Chief Technical Office +44(0)2034687000
> Veber: The Hosting Specialists Fax +44(0)871 522 7057
> http://www.veber.co.uk<http://www.veber.co.uk/>
>
> Follow us on Twitter: www.twitter.com/veberhost<http://www.twitter.com/veberhost>
> Follow us on Facebook: www.facebook.com/veberhost<http://www.facebook.com/veberhost>
>